2023-02-14 15:58:45,077 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0 2023-02-14 15:58:45,090 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-14 15:58:45,104 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-14 15:58:45,104 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0, deleteOnExit=true 2023-02-14 15:58:45,104 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-14 15:58:45,105 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/test.cache.data in system properties and HBase conf 2023-02-14 15:58:45,105 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/hadoop.tmp.dir in system properties and HBase conf 2023-02-14 15:58:45,106 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/hadoop.log.dir in system properties and HBase conf 2023-02-14 15:58:45,106 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-14 15:58:45,107 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-14 15:58:45,107 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-14 15:58:45,227 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-14 15:58:45,624 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-14 15:58:45,628 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-14 15:58:45,629 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-14 15:58:45,629 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-14 15:58:45,630 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-14 15:58:45,630 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-14 15:58:45,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-14 15:58:45,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-14 15:58:45,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-14 15:58:45,632 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-14 15:58:45,632 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/nfs.dump.dir in system properties and HBase conf 2023-02-14 15:58:45,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/java.io.tmpdir in system properties and HBase conf 2023-02-14 15:58:45,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-14 15:58:45,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-14 15:58:45,634 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-14 15:58:46,185 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-14 15:58:46,189 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-14 15:58:46,485 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-14 15:58:46,660 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-14 15:58:46,679 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 15:58:46,715 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 15:58:46,746 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/java.io.tmpdir/Jetty_localhost_44393_hdfs____.cxb5u1/webapp 2023-02-14 15:58:46,899 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44393 2023-02-14 15:58:46,953 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-14 15:58:46,953 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-14 15:58:47,343 WARN [Listener at localhost/43519] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 15:58:47,453 WARN [Listener at localhost/43519] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 15:58:47,471 WARN [Listener at localhost/43519] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 15:58:47,477 INFO [Listener at localhost/43519] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 15:58:47,483 INFO [Listener at localhost/43519] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/java.io.tmpdir/Jetty_localhost_44637_datanode____.14cwvz/webapp 2023-02-14 15:58:47,610 INFO [Listener at localhost/43519] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44637 2023-02-14 15:58:47,973 WARN [Listener at localhost/43055] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 15:58:48,029 WARN [Listener at localhost/43055] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 15:58:48,040 WARN [Listener at localhost/43055] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 15:58:48,042 INFO [Listener at localhost/43055] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 15:58:48,049 INFO [Listener at localhost/43055] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/java.io.tmpdir/Jetty_localhost_33367_datanode____r4frgb/webapp 2023-02-14 15:58:48,175 INFO [Listener at localhost/43055] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33367 2023-02-14 15:58:48,188 WARN [Listener at localhost/41245] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 15:58:48,218 WARN [Listener at localhost/41245] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 15:58:48,221 WARN [Listener at localhost/41245] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 15:58:48,223 INFO [Listener at localhost/41245] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 15:58:48,228 INFO [Listener at localhost/41245] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/java.io.tmpdir/Jetty_localhost_35237_datanode____adya8r/webapp 2023-02-14 15:58:48,338 INFO [Listener at localhost/41245] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35237 2023-02-14 15:58:48,348 WARN [Listener at localhost/39279] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 15:58:48,519 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcfef802814bd978a: Processing first storage report for DS-20035491-f562-4f17-bdfb-c116b57f71dc from datanode 10989456-81c2-411f-a52d-a50290e1d258 2023-02-14 15:58:48,521 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcfef802814bd978a: from storage DS-20035491-f562-4f17-bdfb-c116b57f71dc node DatanodeRegistration(127.0.0.1:33235, datanodeUuid=10989456-81c2-411f-a52d-a50290e1d258, infoPort=37283, infoSecurePort=0, ipcPort=39279, storageInfo=lv=-57;cid=testClusterID;nsid=1966096728;c=1676390326263), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-14 15:58:48,521 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x202263383d7ccccd: Processing first storage report for DS-bb53f587-028a-4a1b-b339-e33daa800a32 from datanode a5a6307c-b6e1-4088-b0f5-35057fac4da6 2023-02-14 15:58:48,521 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x202263383d7ccccd: from storage DS-bb53f587-028a-4a1b-b339-e33daa800a32 node DatanodeRegistration(127.0.0.1:33527, datanodeUuid=a5a6307c-b6e1-4088-b0f5-35057fac4da6, infoPort=32985, infoSecurePort=0, ipcPort=43055, storageInfo=lv=-57;cid=testClusterID;nsid=1966096728;c=1676390326263), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 15:58:48,521 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd99c83e916c4bb62: Processing first storage report for DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d from datanode ce1697e0-c232-403e-a1bb-879864aa955f 2023-02-14 15:58:48,522 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd99c83e916c4bb62: from storage DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d node DatanodeRegistration(127.0.0.1:43343, datanodeUuid=ce1697e0-c232-403e-a1bb-879864aa955f, infoPort=38259, infoSecurePort=0, ipcPort=41245, storageInfo=lv=-57;cid=testClusterID;nsid=1966096728;c=1676390326263), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-14 15:58:48,522 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcfef802814bd978a: Processing first storage report for DS-ae401269-e1d6-421b-85c2-c8b6a0880064 from datanode 10989456-81c2-411f-a52d-a50290e1d258 2023-02-14 15:58:48,522 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcfef802814bd978a: from storage DS-ae401269-e1d6-421b-85c2-c8b6a0880064 node DatanodeRegistration(127.0.0.1:33235, datanodeUuid=10989456-81c2-411f-a52d-a50290e1d258, infoPort=37283, infoSecurePort=0, ipcPort=39279, storageInfo=lv=-57;cid=testClusterID;nsid=1966096728;c=1676390326263), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 15:58:48,522 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x202263383d7ccccd: Processing first storage report for DS-6a11501e-a6ab-488d-beef-9ee207f61dff from datanode a5a6307c-b6e1-4088-b0f5-35057fac4da6 2023-02-14 15:58:48,523 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x202263383d7ccccd: from storage DS-6a11501e-a6ab-488d-beef-9ee207f61dff node DatanodeRegistration(127.0.0.1:33527, datanodeUuid=a5a6307c-b6e1-4088-b0f5-35057fac4da6, infoPort=32985, infoSecurePort=0, ipcPort=43055, storageInfo=lv=-57;cid=testClusterID;nsid=1966096728;c=1676390326263), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 15:58:48,523 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd99c83e916c4bb62: Processing first storage report for DS-76ebfc26-09b4-49e2-90e1-f7255e84810a from datanode ce1697e0-c232-403e-a1bb-879864aa955f 2023-02-14 15:58:48,523 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd99c83e916c4bb62: from storage DS-76ebfc26-09b4-49e2-90e1-f7255e84810a node DatanodeRegistration(127.0.0.1:43343, datanodeUuid=ce1697e0-c232-403e-a1bb-879864aa955f, infoPort=38259, infoSecurePort=0, ipcPort=41245, storageInfo=lv=-57;cid=testClusterID;nsid=1966096728;c=1676390326263), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 15:58:48,914 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0 2023-02-14 15:58:48,982 INFO [Listener at localhost/39279] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/zookeeper_0, clientPort=55782, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-14 15:58:48,994 INFO [Listener at localhost/39279] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55782 2023-02-14 15:58:49,001 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:49,003 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:49,673 INFO [Listener at localhost/39279] util.FSUtils(479): Created version file at hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24 with version=8 2023-02-14 15:58:49,673 INFO [Listener at localhost/39279] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/hbase-staging 2023-02-14 15:58:50,016 INFO [Listener at localhost/39279] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-14 15:58:50,505 INFO [Listener at localhost/39279] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-14 15:58:50,537 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:50,537 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:50,538 INFO [Listener at localhost/39279] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 15:58:50,538 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:50,538 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 15:58:50,675 INFO [Listener at localhost/39279] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 15:58:50,760 DEBUG [Listener at localhost/39279] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-14 15:58:50,866 INFO [Listener at localhost/39279] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33899 2023-02-14 15:58:50,876 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:50,878 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:50,898 INFO [Listener at localhost/39279] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33899 connecting to ZooKeeper ensemble=127.0.0.1:55782 2023-02-14 15:58:50,941 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:338990x0, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 15:58:50,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:33899-0x100538e544d0000 connected 2023-02-14 15:58:50,967 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 15:58:50,968 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:58:50,971 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 15:58:50,982 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33899 2023-02-14 15:58:50,982 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33899 2023-02-14 15:58:50,983 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33899 2023-02-14 15:58:50,983 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33899 2023-02-14 15:58:50,984 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33899 2023-02-14 15:58:50,990 INFO [Listener at localhost/39279] master.HMaster(444): hbase.rootdir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24, hbase.cluster.distributed=false 2023-02-14 15:58:51,070 INFO [Listener at localhost/39279] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-14 15:58:51,070 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,071 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,071 INFO [Listener at localhost/39279] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 15:58:51,071 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,071 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 15:58:51,076 INFO [Listener at localhost/39279] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 15:58:51,079 INFO [Listener at localhost/39279] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43707 2023-02-14 15:58:51,081 INFO [Listener at localhost/39279] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 15:58:51,086 INFO [Listener at localhost/39279] mob.MobFileCache(124): MobFileCache disabled 2023-02-14 15:58:51,088 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,089 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,091 INFO [Listener at localhost/39279] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43707 connecting to ZooKeeper ensemble=127.0.0.1:55782 2023-02-14 15:58:51,094 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:437070x0, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 15:58:51,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43707-0x100538e544d0001 connected 2023-02-14 15:58:51,096 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 15:58:51,097 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:58:51,098 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 15:58:51,100 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43707 2023-02-14 15:58:51,100 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43707 2023-02-14 15:58:51,101 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43707 2023-02-14 15:58:51,101 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43707 2023-02-14 15:58:51,101 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43707 2023-02-14 15:58:51,114 INFO [Listener at localhost/39279] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-14 15:58:51,114 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,115 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,115 INFO [Listener at localhost/39279] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 15:58:51,115 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,116 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 15:58:51,116 INFO [Listener at localhost/39279] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 15:58:51,117 INFO [Listener at localhost/39279] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40531 2023-02-14 15:58:51,118 INFO [Listener at localhost/39279] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 15:58:51,119 INFO [Listener at localhost/39279] mob.MobFileCache(124): MobFileCache disabled 2023-02-14 15:58:51,120 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,122 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,124 INFO [Listener at localhost/39279] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40531 connecting to ZooKeeper ensemble=127.0.0.1:55782 2023-02-14 15:58:51,128 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:405310x0, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 15:58:51,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40531-0x100538e544d0002 connected 2023-02-14 15:58:51,129 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 15:58:51,130 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:58:51,131 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 15:58:51,132 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40531 2023-02-14 15:58:51,133 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40531 2023-02-14 15:58:51,133 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40531 2023-02-14 15:58:51,135 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40531 2023-02-14 15:58:51,135 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40531 2023-02-14 15:58:51,147 INFO [Listener at localhost/39279] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-14 15:58:51,147 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,147 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,148 INFO [Listener at localhost/39279] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 15:58:51,148 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 15:58:51,148 INFO [Listener at localhost/39279] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 15:58:51,148 INFO [Listener at localhost/39279] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 15:58:51,150 INFO [Listener at localhost/39279] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37877 2023-02-14 15:58:51,151 INFO [Listener at localhost/39279] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 15:58:51,151 INFO [Listener at localhost/39279] mob.MobFileCache(124): MobFileCache disabled 2023-02-14 15:58:51,152 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,155 INFO [Listener at localhost/39279] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,157 INFO [Listener at localhost/39279] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37877 connecting to ZooKeeper ensemble=127.0.0.1:55782 2023-02-14 15:58:51,162 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:378770x0, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 15:58:51,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37877-0x100538e544d0003 connected 2023-02-14 15:58:51,163 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 15:58:51,164 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:58:51,165 DEBUG [Listener at localhost/39279] zookeeper.ZKUtil(164): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 15:58:51,166 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37877 2023-02-14 15:58:51,166 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37877 2023-02-14 15:58:51,166 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37877 2023-02-14 15:58:51,166 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37877 2023-02-14 15:58:51,167 DEBUG [Listener at localhost/39279] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37877 2023-02-14 15:58:51,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:51,178 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 15:58:51,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:51,201 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 15:58:51,202 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 15:58:51,202 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:51,202 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 15:58:51,201 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 15:58:51,204 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 15:58:51,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,33899,1676390329846 from backup master directory 2023-02-14 15:58:51,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 15:58:51,210 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:51,210 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 15:58:51,211 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 15:58:51,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:51,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-14 15:58:51,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-14 15:58:51,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/hbase.id with ID: c27f290d-d6a4-4dad-a27e-0207d919b47b 2023-02-14 15:58:51,371 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 15:58:51,397 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:51,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4325d116 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:51,527 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@466d082f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:51,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 15:58:51,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-14 15:58:51,569 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-14 15:58:51,569 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-14 15:58:51,571 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-14 15:58:51,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-14 15:58:51,578 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 15:58:51,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store-tmp 2023-02-14 15:58:51,676 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:51,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 15:58:51,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:58:51,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:58:51,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 15:58:51,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:58:51,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:58:51,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-14 15:58:51,679 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/WALs/jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:51,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33899%2C1676390329846, suffix=, logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/WALs/jenkins-hbase4.apache.org,33899,1676390329846, archiveDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/oldWALs, maxLogs=10 2023-02-14 15:58:51,750 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK] 2023-02-14 15:58:51,750 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK] 2023-02-14 15:58:51,750 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK] 2023-02-14 15:58:51,758 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 15:58:51,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/WALs/jenkins-hbase4.apache.org,33899,1676390329846/jenkins-hbase4.apache.org%2C33899%2C1676390329846.1676390331708 2023-02-14 15:58:51,827 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK], DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK], DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK]] 2023-02-14 15:58:51,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-14 15:58:51,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:51,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 15:58:51,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 15:58:51,889 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-14 15:58:51,903 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-14 15:58:51,929 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-14 15:58:51,942 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:51,949 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-14 15:58:51,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-14 15:58:51,964 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 15:58:51,968 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:51,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65523870, jitterRate=-0.023618251085281372}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:51,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-14 15:58:51,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-14 15:58:51,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-14 15:58:51,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-14 15:58:51,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-14 15:58:51,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-14 15:58:52,026 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-14 15:58:52,026 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-14 15:58:52,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-14 15:58:52,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-14 15:58:52,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-14 15:58:52,082 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-14 15:58:52,083 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-14 15:58:52,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-14 15:58:52,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-14 15:58:52,095 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:52,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-14 15:58:52,097 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-14 15:58:52,110 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-14 15:58:52,115 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 15:58:52,116 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 15:58:52,116 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 15:58:52,116 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:52,116 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 15:58:52,117 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,33899,1676390329846, sessionid=0x100538e544d0000, setting cluster-up flag (Was=false) 2023-02-14 15:58:52,134 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:52,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-14 15:58:52,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:52,146 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:52,152 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-14 15:58:52,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:52,155 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp 2023-02-14 15:58:52,171 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(952): ClusterId : c27f290d-d6a4-4dad-a27e-0207d919b47b 2023-02-14 15:58:52,171 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(952): ClusterId : c27f290d-d6a4-4dad-a27e-0207d919b47b 2023-02-14 15:58:52,171 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(952): ClusterId : c27f290d-d6a4-4dad-a27e-0207d919b47b 2023-02-14 15:58:52,177 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 15:58:52,177 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 15:58:52,177 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 15:58:52,183 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 15:58:52,183 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 15:58:52,183 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 15:58:52,183 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 15:58:52,183 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 15:58:52,183 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 15:58:52,188 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 15:58:52,188 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 15:58:52,188 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 15:58:52,189 DEBUG [RS:2;jenkins-hbase4:37877] zookeeper.ReadOnlyZKClient(139): Connect 0x361a42ab to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:52,190 DEBUG [RS:0;jenkins-hbase4:43707] zookeeper.ReadOnlyZKClient(139): Connect 0x38688fe2 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:52,190 DEBUG [RS:1;jenkins-hbase4:40531] zookeeper.ReadOnlyZKClient(139): Connect 0x0033bdb6 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:52,201 DEBUG [RS:2;jenkins-hbase4:37877] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38ddba18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:52,201 DEBUG [RS:0;jenkins-hbase4:43707] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36af669f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:52,201 DEBUG [RS:2;jenkins-hbase4:37877] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ce229db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-14 15:58:52,202 DEBUG [RS:0;jenkins-hbase4:43707] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3290e492, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-14 15:58:52,202 DEBUG [RS:1;jenkins-hbase4:40531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ef453c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:52,202 DEBUG [RS:1;jenkins-hbase4:40531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@acbd6c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-14 15:58:52,225 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:40531 2023-02-14 15:58:52,228 INFO [RS:1;jenkins-hbase4:40531] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 15:58:52,228 INFO [RS:1;jenkins-hbase4:40531] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 15:58:52,228 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 15:58:52,229 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43707 2023-02-14 15:58:52,230 INFO [RS:0;jenkins-hbase4:43707] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 15:58:52,230 INFO [RS:0;jenkins-hbase4:43707] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 15:58:52,230 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 15:58:52,231 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:37877 2023-02-14 15:58:52,231 INFO [RS:2;jenkins-hbase4:37877] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 15:58:52,231 INFO [RS:2;jenkins-hbase4:37877] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 15:58:52,231 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 15:58:52,233 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33899,1676390329846 with isa=jenkins-hbase4.apache.org/172.31.14.131:40531, startcode=1676390331113 2023-02-14 15:58:52,233 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33899,1676390329846 with isa=jenkins-hbase4.apache.org/172.31.14.131:37877, startcode=1676390331146 2023-02-14 15:58:52,233 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33899,1676390329846 with isa=jenkins-hbase4.apache.org/172.31.14.131:43707, startcode=1676390331069 2023-02-14 15:58:52,251 DEBUG [RS:0;jenkins-hbase4:43707] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 15:58:52,251 DEBUG [RS:1;jenkins-hbase4:40531] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 15:58:52,251 DEBUG [RS:2;jenkins-hbase4:37877] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 15:58:52,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-14 15:58:52,319 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41015, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 15:58:52,319 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40903, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 15:58:52,319 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59001, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 15:58:52,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-14 15:58:52,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-14 15:58:52,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-14 15:58:52,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-14 15:58:52,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-14 15:58:52,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-14 15:58:52,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676390362323 2023-02-14 15:58:52,325 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-14 15:58:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33899] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:58:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:58:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33899] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:58:52,330 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-14 15:58:52,330 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-14 15:58:52,335 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-14 15:58:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-14 15:58:52,345 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-14 15:58:52,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-14 15:58:52,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-14 15:58:52,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-14 15:58:52,347 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-14 15:58:52,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-14 15:58:52,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-14 15:58:52,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-14 15:58:52,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-14 15:58:52,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676390332355,5,FailOnTimeoutGroup] 2023-02-14 15:58:52,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676390332355,5,FailOnTimeoutGroup] 2023-02-14 15:58:52,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-14 15:58:52,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,360 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 15:58:52,360 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 15:58:52,361 WARN [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-14 15:58:52,360 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 15:58:52,361 WARN [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-14 15:58:52,361 WARN [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-14 15:58:52,387 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-14 15:58:52,388 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-14 15:58:52,389 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24 2023-02-14 15:58:52,415 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:52,417 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 15:58:52,420 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/info 2023-02-14 15:58:52,421 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 15:58:52,422 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:52,422 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 15:58:52,426 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/rep_barrier 2023-02-14 15:58:52,426 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 15:58:52,427 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:52,427 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 15:58:52,430 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/table 2023-02-14 15:58:52,430 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 15:58:52,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:52,433 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740 2023-02-14 15:58:52,434 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740 2023-02-14 15:58:52,437 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 15:58:52,439 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 15:58:52,443 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:52,444 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69838562, jitterRate=0.04067566990852356}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 15:58:52,444 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-14 15:58:52,444 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 15:58:52,444 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 15:58:52,444 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 15:58:52,444 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 15:58:52,444 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 15:58:52,445 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 15:58:52,446 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-14 15:58:52,452 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-14 15:58:52,452 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-14 15:58:52,461 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-14 15:58:52,462 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33899,1676390329846 with isa=jenkins-hbase4.apache.org/172.31.14.131:40531, startcode=1676390331113 2023-02-14 15:58:52,462 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33899,1676390329846 with isa=jenkins-hbase4.apache.org/172.31.14.131:43707, startcode=1676390331069 2023-02-14 15:58:52,462 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33899,1676390329846 with isa=jenkins-hbase4.apache.org/172.31.14.131:37877, startcode=1676390331146 2023-02-14 15:58:52,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33899] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,469 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33899] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,476 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-14 15:58:52,477 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24 2023-02-14 15:58:52,477 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43519 2023-02-14 15:58:52,477 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 15:58:52,482 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24 2023-02-14 15:58:52,482 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43519 2023-02-14 15:58:52,482 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 15:58:52,483 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24 2023-02-14 15:58:52,483 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43519 2023-02-14 15:58:52,483 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 15:58:52,484 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 15:58:52,486 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-14 15:58:52,489 DEBUG [RS:0;jenkins-hbase4:43707] zookeeper.ZKUtil(162): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,489 WARN [RS:0;jenkins-hbase4:43707] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 15:58:52,489 INFO [RS:0;jenkins-hbase4:43707] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 15:58:52,489 DEBUG [RS:1;jenkins-hbase4:40531] zookeeper.ZKUtil(162): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,489 DEBUG [RS:2;jenkins-hbase4:37877] zookeeper.ZKUtil(162): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,490 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37877,1676390331146] 2023-02-14 15:58:52,490 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43707,1676390331069] 2023-02-14 15:58:52,490 WARN [RS:2;jenkins-hbase4:37877] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 15:58:52,490 WARN [RS:1;jenkins-hbase4:40531] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 15:58:52,490 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,490 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40531,1676390331113] 2023-02-14 15:58:52,490 INFO [RS:1;jenkins-hbase4:40531] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 15:58:52,490 INFO [RS:2;jenkins-hbase4:37877] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 15:58:52,492 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,493 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,503 DEBUG [RS:0;jenkins-hbase4:43707] zookeeper.ZKUtil(162): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,503 DEBUG [RS:1;jenkins-hbase4:40531] zookeeper.ZKUtil(162): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,503 DEBUG [RS:2;jenkins-hbase4:37877] zookeeper.ZKUtil(162): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,504 DEBUG [RS:0;jenkins-hbase4:43707] zookeeper.ZKUtil(162): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,504 DEBUG [RS:1;jenkins-hbase4:40531] zookeeper.ZKUtil(162): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,504 DEBUG [RS:2;jenkins-hbase4:37877] zookeeper.ZKUtil(162): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,504 DEBUG [RS:0;jenkins-hbase4:43707] zookeeper.ZKUtil(162): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,505 DEBUG [RS:1;jenkins-hbase4:40531] zookeeper.ZKUtil(162): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,505 DEBUG [RS:2;jenkins-hbase4:37877] zookeeper.ZKUtil(162): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,514 DEBUG [RS:0;jenkins-hbase4:43707] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 15:58:52,514 DEBUG [RS:2;jenkins-hbase4:37877] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 15:58:52,514 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 15:58:52,523 INFO [RS:0;jenkins-hbase4:43707] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 15:58:52,523 INFO [RS:1;jenkins-hbase4:40531] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 15:58:52,523 INFO [RS:2;jenkins-hbase4:37877] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 15:58:52,544 INFO [RS:1;jenkins-hbase4:40531] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 15:58:52,544 INFO [RS:0;jenkins-hbase4:43707] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 15:58:52,544 INFO [RS:2;jenkins-hbase4:37877] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 15:58:52,547 INFO [RS:0;jenkins-hbase4:43707] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 15:58:52,547 INFO [RS:1;jenkins-hbase4:40531] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 15:58:52,547 INFO [RS:2;jenkins-hbase4:37877] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 15:58:52,548 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,548 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,548 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,549 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 15:58:52,549 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 15:58:52,549 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 15:58:52,557 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,557 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,557 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,557 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,557 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-14 15:58:52,558 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-14 15:58:52,559 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,558 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:1;jenkins-hbase4:40531] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:0;jenkins-hbase4:43707] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,559 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,560 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,560 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-14 15:58:52,560 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,560 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,560 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,561 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,561 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,561 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,561 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,560 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,561 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,561 DEBUG [RS:2;jenkins-hbase4:37877] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-14 15:58:52,562 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,562 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,562 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,582 INFO [RS:2;jenkins-hbase4:37877] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 15:58:52,582 INFO [RS:0;jenkins-hbase4:43707] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 15:58:52,582 INFO [RS:1;jenkins-hbase4:40531] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 15:58:52,585 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37877,1676390331146-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,585 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40531,1676390331113-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,585 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43707,1676390331069-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:52,606 INFO [RS:1;jenkins-hbase4:40531] regionserver.Replication(203): jenkins-hbase4.apache.org,40531,1676390331113 started 2023-02-14 15:58:52,606 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40531,1676390331113, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40531, sessionid=0x100538e544d0002 2023-02-14 15:58:52,610 INFO [RS:2;jenkins-hbase4:37877] regionserver.Replication(203): jenkins-hbase4.apache.org,37877,1676390331146 started 2023-02-14 15:58:52,610 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37877,1676390331146, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37877, sessionid=0x100538e544d0003 2023-02-14 15:58:52,622 INFO [RS:0;jenkins-hbase4:43707] regionserver.Replication(203): jenkins-hbase4.apache.org,43707,1676390331069 started 2023-02-14 15:58:52,622 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43707,1676390331069, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43707, sessionid=0x100538e544d0001 2023-02-14 15:58:52,622 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 15:58:52,623 DEBUG [RS:0;jenkins-hbase4:43707] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,623 DEBUG [RS:0;jenkins-hbase4:43707] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43707,1676390331069' 2023-02-14 15:58:52,623 DEBUG [RS:0;jenkins-hbase4:43707] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 15:58:52,624 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 15:58:52,624 DEBUG [RS:2;jenkins-hbase4:37877] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,624 DEBUG [RS:2;jenkins-hbase4:37877] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37877,1676390331146' 2023-02-14 15:58:52,625 DEBUG [RS:2;jenkins-hbase4:37877] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 15:58:52,630 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 15:58:52,631 DEBUG [RS:1;jenkins-hbase4:40531] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,631 DEBUG [RS:1;jenkins-hbase4:40531] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40531,1676390331113' 2023-02-14 15:58:52,631 DEBUG [RS:1;jenkins-hbase4:40531] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 15:58:52,631 DEBUG [RS:0;jenkins-hbase4:43707] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 15:58:52,631 DEBUG [RS:2;jenkins-hbase4:37877] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 15:58:52,632 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 15:58:52,632 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 15:58:52,632 DEBUG [RS:0;jenkins-hbase4:43707] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:52,632 DEBUG [RS:0;jenkins-hbase4:43707] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43707,1676390331069' 2023-02-14 15:58:52,632 DEBUG [RS:0;jenkins-hbase4:43707] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 15:58:52,633 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 15:58:52,633 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 15:58:52,633 DEBUG [RS:1;jenkins-hbase4:40531] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 15:58:52,635 DEBUG [RS:0;jenkins-hbase4:43707] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 15:58:52,633 DEBUG [RS:2;jenkins-hbase4:37877] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:52,635 DEBUG [RS:2;jenkins-hbase4:37877] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37877,1676390331146' 2023-02-14 15:58:52,635 DEBUG [RS:2;jenkins-hbase4:37877] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 15:58:52,636 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 15:58:52,636 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 15:58:52,636 DEBUG [RS:1;jenkins-hbase4:40531] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,636 DEBUG [RS:0;jenkins-hbase4:43707] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 15:58:52,636 DEBUG [RS:1;jenkins-hbase4:40531] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40531,1676390331113' 2023-02-14 15:58:52,636 INFO [RS:0;jenkins-hbase4:43707] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 15:58:52,636 DEBUG [RS:1;jenkins-hbase4:40531] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 15:58:52,636 INFO [RS:0;jenkins-hbase4:43707] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 15:58:52,636 DEBUG [RS:2;jenkins-hbase4:37877] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 15:58:52,638 DEBUG [jenkins-hbase4:33899] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-14 15:58:52,639 DEBUG [RS:1;jenkins-hbase4:40531] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 15:58:52,639 DEBUG [RS:2;jenkins-hbase4:37877] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 15:58:52,641 DEBUG [RS:1;jenkins-hbase4:40531] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 15:58:52,641 INFO [RS:1;jenkins-hbase4:40531] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 15:58:52,641 INFO [RS:1;jenkins-hbase4:40531] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 15:58:52,644 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:58:52,639 INFO [RS:2;jenkins-hbase4:37877] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 15:58:52,644 INFO [RS:2;jenkins-hbase4:37877] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 15:58:52,652 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:58:52,652 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:58:52,652 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:58:52,652 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:58:52,656 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40531,1676390331113, state=OPENING 2023-02-14 15:58:52,664 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-14 15:58:52,666 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:52,666 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 15:58:52,669 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:52,752 INFO [RS:2;jenkins-hbase4:37877] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37877%2C1676390331146, suffix=, logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,37877,1676390331146, archiveDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs, maxLogs=32 2023-02-14 15:58:52,752 INFO [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40531%2C1676390331113, suffix=, logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,40531,1676390331113, archiveDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs, maxLogs=32 2023-02-14 15:58:52,762 INFO [RS:0;jenkins-hbase4:43707] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43707%2C1676390331069, suffix=, logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,43707,1676390331069, archiveDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs, maxLogs=32 2023-02-14 15:58:52,817 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK] 2023-02-14 15:58:52,824 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK] 2023-02-14 15:58:52,826 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK] 2023-02-14 15:58:52,826 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK] 2023-02-14 15:58:52,826 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK] 2023-02-14 15:58:52,831 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK] 2023-02-14 15:58:52,833 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK] 2023-02-14 15:58:52,834 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK] 2023-02-14 15:58:52,834 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK] 2023-02-14 15:58:52,842 INFO [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,40531,1676390331113/jenkins-hbase4.apache.org%2C40531%2C1676390331113.1676390332761 2023-02-14 15:58:52,842 INFO [RS:0;jenkins-hbase4:43707] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,43707,1676390331069/jenkins-hbase4.apache.org%2C43707%2C1676390331069.1676390332764 2023-02-14 15:58:52,842 DEBUG [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK], DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK], DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK]] 2023-02-14 15:58:52,843 DEBUG [RS:0;jenkins-hbase4:43707] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK], DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK], DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK]] 2023-02-14 15:58:52,844 INFO [RS:2;jenkins-hbase4:37877] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,37877,1676390331146/jenkins-hbase4.apache.org%2C37877%2C1676390331146.1676390332761 2023-02-14 15:58:52,844 DEBUG [RS:2;jenkins-hbase4:37877] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK], DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK], DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK]] 2023-02-14 15:58:52,869 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:52,872 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 15:58:52,874 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35110, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 15:58:52,885 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-14 15:58:52,885 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 15:58:52,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40531%2C1676390331113.meta, suffix=.meta, logDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,40531,1676390331113, archiveDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs, maxLogs=32 2023-02-14 15:58:52,911 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK] 2023-02-14 15:58:52,912 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK] 2023-02-14 15:58:52,913 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK] 2023-02-14 15:58:52,918 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/WALs/jenkins-hbase4.apache.org,40531,1676390331113/jenkins-hbase4.apache.org%2C40531%2C1676390331113.meta.1676390332891.meta 2023-02-14 15:58:52,918 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33527,DS-bb53f587-028a-4a1b-b339-e33daa800a32,DISK], DatanodeInfoWithStorage[127.0.0.1:43343,DS-1a3d677a-5dfd-4d72-a4a2-d92d4e56630d,DISK], DatanodeInfoWithStorage[127.0.0.1:33235,DS-20035491-f562-4f17-bdfb-c116b57f71dc,DISK]] 2023-02-14 15:58:52,918 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-14 15:58:52,920 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-14 15:58:52,937 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-14 15:58:52,942 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-14 15:58:52,947 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-14 15:58:52,947 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:52,947 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-14 15:58:52,947 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-14 15:58:52,950 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 15:58:52,952 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/info 2023-02-14 15:58:52,952 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/info 2023-02-14 15:58:52,952 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 15:58:52,953 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:52,953 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 15:58:52,955 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/rep_barrier 2023-02-14 15:58:52,955 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/rep_barrier 2023-02-14 15:58:52,955 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 15:58:52,956 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:52,956 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 15:58:52,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/table 2023-02-14 15:58:52,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/table 2023-02-14 15:58:52,958 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 15:58:52,959 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:52,961 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740 2023-02-14 15:58:52,964 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740 2023-02-14 15:58:52,967 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 15:58:52,972 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 15:58:52,974 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62509401, jitterRate=-0.06853733956813812}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 15:58:52,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-14 15:58:52,984 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676390332857 2023-02-14 15:58:53,000 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-14 15:58:53,001 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-14 15:58:53,002 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40531,1676390331113, state=OPEN 2023-02-14 15:58:53,006 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-14 15:58:53,006 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 15:58:53,010 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-14 15:58:53,010 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40531,1676390331113 in 337 msec 2023-02-14 15:58:53,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-14 15:58:53,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 551 msec 2023-02-14 15:58:53,021 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 804 msec 2023-02-14 15:58:53,022 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1676390333022, completionTime=-1 2023-02-14 15:58:53,022 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-14 15:58:53,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-14 15:58:53,097 DEBUG [hconnection-0x5c7402a7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:58:53,099 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35120, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:58:53,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-14 15:58:53,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676390393115 2023-02-14 15:58:53,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676390453115 2023-02-14 15:58:53,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 93 msec 2023-02-14 15:58:53,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33899,1676390329846-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:53,138 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33899,1676390329846-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:53,138 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33899,1676390329846-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:53,139 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:33899, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:53,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-14 15:58:53,149 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-14 15:58:53,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-14 15:58:53,159 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-14 15:58:53,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-14 15:58:53,171 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 15:58:53,173 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 15:58:53,197 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,199 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3 empty. 2023-02-14 15:58:53,200 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,200 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-14 15:58:53,230 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-14 15:58:53,233 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 151fcdb2b3eadef830a0e082099efff3, NAME => 'hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:53,253 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:53,253 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 151fcdb2b3eadef830a0e082099efff3, disabling compactions & flushes 2023-02-14 15:58:53,253 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,253 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,253 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. after waiting 0 ms 2023-02-14 15:58:53,253 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,253 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,253 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 151fcdb2b3eadef830a0e082099efff3: 2023-02-14 15:58:53,258 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 15:58:53,272 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676390333260"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390333260"}]},"ts":"1676390333260"} 2023-02-14 15:58:53,301 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 15:58:53,302 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 15:58:53,307 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390333303"}]},"ts":"1676390333303"} 2023-02-14 15:58:53,311 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-14 15:58:53,316 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:58:53,317 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:58:53,317 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:58:53,317 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:58:53,317 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:58:53,319 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=151fcdb2b3eadef830a0e082099efff3, ASSIGN}] 2023-02-14 15:58:53,322 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=151fcdb2b3eadef830a0e082099efff3, ASSIGN 2023-02-14 15:58:53,324 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=151fcdb2b3eadef830a0e082099efff3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:58:53,476 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 15:58:53,478 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=151fcdb2b3eadef830a0e082099efff3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:53,478 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676390333477"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390333477"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390333477"}]},"ts":"1676390333477"} 2023-02-14 15:58:53,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 151fcdb2b3eadef830a0e082099efff3, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:53,650 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,650 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 151fcdb2b3eadef830a0e082099efff3, NAME => 'hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.', STARTKEY => '', ENDKEY => ''} 2023-02-14 15:58:53,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:53,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,653 INFO [StoreOpener-151fcdb2b3eadef830a0e082099efff3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,656 DEBUG [StoreOpener-151fcdb2b3eadef830a0e082099efff3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/info 2023-02-14 15:58:53,656 DEBUG [StoreOpener-151fcdb2b3eadef830a0e082099efff3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/info 2023-02-14 15:58:53,657 INFO [StoreOpener-151fcdb2b3eadef830a0e082099efff3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 151fcdb2b3eadef830a0e082099efff3 columnFamilyName info 2023-02-14 15:58:53,657 INFO [StoreOpener-151fcdb2b3eadef830a0e082099efff3-1] regionserver.HStore(310): Store=151fcdb2b3eadef830a0e082099efff3/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:53,659 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,660 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,665 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:58:53,669 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:53,670 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 151fcdb2b3eadef830a0e082099efff3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74136010, jitterRate=0.10471263527870178}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:53,670 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 151fcdb2b3eadef830a0e082099efff3: 2023-02-14 15:58:53,677 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3., pid=6, masterSystemTime=1676390333640 2023-02-14 15:58:53,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,684 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:58:53,686 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=151fcdb2b3eadef830a0e082099efff3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:53,686 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676390333685"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390333685"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390333685"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390333685"}]},"ts":"1676390333685"} 2023-02-14 15:58:53,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-14 15:58:53,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 151fcdb2b3eadef830a0e082099efff3, server=jenkins-hbase4.apache.org,40531,1676390331113 in 207 msec 2023-02-14 15:58:53,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-14 15:58:53,699 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=151fcdb2b3eadef830a0e082099efff3, ASSIGN in 375 msec 2023-02-14 15:58:53,700 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 15:58:53,701 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390333700"}]},"ts":"1676390333700"} 2023-02-14 15:58:53,705 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-14 15:58:53,709 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 15:58:53,713 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 549 msec 2023-02-14 15:58:53,771 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-14 15:58:53,772 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-14 15:58:53,772 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:58:53,809 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-14 15:58:53,829 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-14 15:58:53,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 36 msec 2023-02-14 15:58:53,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-14 15:58:53,860 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-14 15:58:53,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-02-14 15:58:53,891 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-14 15:58:53,894 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-14 15:58:53,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.683sec 2023-02-14 15:58:53,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-14 15:58:53,933 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-14 15:58:53,934 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-14 15:58:53,935 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33899,1676390329846-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-14 15:58:53,936 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33899,1676390329846-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-14 15:58:53,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-02-14 15:58:53,977 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(139): Connect 0x297db6e1 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:53,982 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c633018, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:53,995 DEBUG [hconnection-0x6de71207-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:58:54,006 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53444, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:58:54,016 INFO [Listener at localhost/39279] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:58:54,049 INFO [Listener at localhost/39279] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=358, OpenFileDescriptor=688, MaxFileDescriptor=60000, SystemLoadAverage=262, ProcessCount=168, AvailableMemoryMB=4294 2023-02-14 15:58:54,066 DEBUG [Listener at localhost/39279] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-14 15:58:54,070 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-14 15:58:54,085 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-14 15:58:54,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:58:54,092 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676390334053 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 15:58:54,095 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676390334053 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 15:58:54,097 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676390334053" procId is: 9 2023-02-14 15:58:54,104 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,105 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,105 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,105 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,105 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,105 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,106 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d empty. 2023-02-14 15:58:54,106 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 empty. 2023-02-14 15:58:54,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 15:58:54,106 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 empty. 2023-02-14 15:58:54,107 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 empty. 2023-02-14 15:58:54,107 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 empty. 2023-02-14 15:58:54,107 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,108 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b empty. 2023-02-14 15:58:54,110 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,111 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,111 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,111 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,111 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,111 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676390334053 regions 2023-02-14 15:58:54,149 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000001 2023-02-14 15:58:54,153 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(7671): creating {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:54,154 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(7671): creating {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:54,154 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(7671): creating {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:54,249 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,249 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:58:54,249 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,249 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,250 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:58:54,251 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,251 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,252 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:58:54,253 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(7671): creating {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:54,253 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:58:54,257 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:58:54,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,257 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:58:54,258 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(7671): creating {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:54,267 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,268 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:58:54,268 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,268 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,268 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:58:54,268 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,268 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,268 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:58:54,269 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(7671): creating {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:58:54,296 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,297 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:58:54,297 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,298 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,298 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:58:54,298 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,298 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,298 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:58:54,305 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,305 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:58:54,305 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,305 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,305 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:58:54,305 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,305 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,305 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-1] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:58:54,309 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,309 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:58:54,309 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,309 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,309 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:58:54,309 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,309 INFO [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,309 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676390334053-pool-2] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:58:54,313 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676390334053 execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 15:58:54,315 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390334314"}]},"ts":"1676390334314"} 2023-02-14 15:58:54,315 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390334314"}]},"ts":"1676390334314"} 2023-02-14 15:58:54,315 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390334314"}]},"ts":"1676390334314"} 2023-02-14 15:58:54,316 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390334314"}]},"ts":"1676390334314"} 2023-02-14 15:58:54,316 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390334314"}]},"ts":"1676390334314"} 2023-02-14 15:58:54,316 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390334314"}]},"ts":"1676390334314"} 2023-02-14 15:58:54,363 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-14 15:58:54,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 15:58:54,366 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676390334053 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 15:58:54,367 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390334366"}]},"ts":"1676390334366"} 2023-02-14 15:58:54,369 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:58:54,375 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:58:54,380 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:58:54,380 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:58:54,380 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:58:54,380 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:58:54,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:58:54,384 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:58:54,385 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:58:54,386 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:58:54,386 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:58:54,389 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:58:54,390 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=false 2023-02-14 15:58:54,390 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=false 2023-02-14 15:58:54,390 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:58:54,391 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=false 2023-02-14 15:58:54,391 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:58:54,396 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:58:54,396 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=false 2023-02-14 15:58:54,539 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:58:54,542 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,542 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,542 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,542 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:54,542 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334542"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334542"}]},"ts":"1676390334542"} 2023-02-14 15:58:54,542 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:54,542 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334542"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334542"}]},"ts":"1676390334542"} 2023-02-14 15:58:54,542 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334542"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334542"}]},"ts":"1676390334542"} 2023-02-14 15:58:54,542 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334542"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334542"}]},"ts":"1676390334542"} 2023-02-14 15:58:54,543 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334542"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334542"}]},"ts":"1676390334542"} 2023-02-14 15:58:54,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=12, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:54,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=11, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:54,548 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:54,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=13, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:54,557 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,557 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334556"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334556"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334556"}]},"ts":"1676390334556"} 2023-02-14 15:58:54,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=14, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:54,559 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=10, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:54,703 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,703 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 15:58:54,707 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46742, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 15:58:54,710 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,710 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 15:58:54,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:58:54,714 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,717 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,717 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:58:54,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,720 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:58:54,720 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:58:54,720 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:58:54,720 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:58:54,721 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,723 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:58:54,723 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:58:54,723 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:58:54,724 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:58:54,725 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:58:54,726 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:54,729 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:54,730 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:54,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:54,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:54,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:54,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:54,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58821407, jitterRate=-0.12349273264408112}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:54,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:58:54,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=17, masterSystemTime=1676390334705 2023-02-14 15:58:54,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:54,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:58:54,754 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:54,754 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334754"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390334754"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390334754"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390334754"}]},"ts":"1676390334754"} 2023-02-14 15:58:54,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,758 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:54,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:54,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61355331, jitterRate=-0.08573432266712189}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:54,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:58:54,763 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:58:54,764 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:58:54,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71968317, jitterRate=0.07241149246692657}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:54,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:58:54,764 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:58:54,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=19, masterSystemTime=1676390334710 2023-02-14 15:58:54,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=18, masterSystemTime=1676390334703 2023-02-14 15:58:54,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=11 2023-02-14 15:58:54,769 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=11, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 211 msec 2023-02-14 15:58:54,769 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:54,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:54,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:58:54,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,773 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,773 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334773"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390334773"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390334773"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390334773"}]},"ts":"1676390334773"} 2023-02-14 15:58:54,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,774 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:54,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,776 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 389 msec 2023-02-14 15:58:54,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:58:54,777 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:54,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,777 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334777"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390334777"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390334777"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390334777"}]},"ts":"1676390334777"} 2023-02-14 15:58:54,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,778 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:58:54,779 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:58:54,779 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:58:54,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:54,781 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,781 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:54,785 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:58:54,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:54,785 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:58:54,786 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:58:54,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68492548, jitterRate=0.0206184983253479}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:54,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:58:54,787 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=13 2023-02-14 15:58:54,788 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=13, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 223 msec 2023-02-14 15:58:54,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=20, masterSystemTime=1676390334705 2023-02-14 15:58:54,790 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:54,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-02-14 15:58:54,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 233 msec 2023-02-14 15:58:54,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,793 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 407 msec 2023-02-14 15:58:54,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:54,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 412 msec 2023-02-14 15:58:54,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:54,799 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:54,799 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334799"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390334799"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390334799"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390334799"}]},"ts":"1676390334799"} 2023-02-14 15:58:54,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:54,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:54,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65655294, jitterRate=-0.021659880876541138}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:54,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:58:54,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=21, masterSystemTime=1676390334710 2023-02-14 15:58:54,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:58:54,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73384579, jitterRate=0.09351544082164764}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:54,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:58:54,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,810 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=14 2023-02-14 15:58:54,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,810 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=14, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 246 msec 2023-02-14 15:58:54,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=16, masterSystemTime=1676390334703 2023-02-14 15:58:54,813 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,813 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334813"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390334813"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390334813"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390334813"}]},"ts":"1676390334813"} 2023-02-14 15:58:54,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 430 msec 2023-02-14 15:58:54,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:54,816 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,816 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334816"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390334816"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390334816"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390334816"}]},"ts":"1676390334816"} 2023-02-14 15:58:54,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=10 2023-02-14 15:58:54,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=10, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 258 msec 2023-02-14 15:58:54,823 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 440 msec 2023-02-14 15:58:54,823 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=12 2023-02-14 15:58:54,824 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 274 msec 2023-02-14 15:58:54,827 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-02-14 15:58:54,827 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 444 msec 2023-02-14 15:58:54,829 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676390334053 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 15:58:54,829 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390334829"}]},"ts":"1676390334829"} 2023-02-14 15:58:54,831 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:58:54,835 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676390334053 execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 15:58:54,838 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676390334053 in 749 msec 2023-02-14 15:58:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 15:58:54,866 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 9 completed 2023-02-14 15:58:54,873 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676390334053 2023-02-14 15:58:54,874 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:54,876 INFO [Listener at localhost/39279] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 15:58:54,898 DEBUG [hconnection-0x6de71207-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:58:54,901 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43622, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:58:54,905 DEBUG [hconnection-0x6de71207-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:58:54,909 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46750, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:58:54,924 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:58:54,929 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:58:54,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:58:54,946 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390334946"}]},"ts":"1676390334946"} 2023-02-14 15:58:54,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-14 15:58:54,948 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:58:54,950 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:58:54,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:58:54,958 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:58:54,958 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:58:54,959 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:58:54,961 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:58:54,961 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,961 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,961 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:54,961 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334961"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334961"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334961"}]},"ts":"1676390334961"} 2023-02-14 15:58:54,962 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334961"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334961"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334961"}]},"ts":"1676390334961"} 2023-02-14 15:58:54,961 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334961"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334961"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334961"}]},"ts":"1676390334961"} 2023-02-14 15:58:54,962 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:54,962 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334962"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334962"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334962"}]},"ts":"1676390334962"} 2023-02-14 15:58:54,963 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:58:54,964 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:54,964 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390334964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334964"}]},"ts":"1676390334964"} 2023-02-14 15:58:54,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=25, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:54,967 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:54,969 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=23, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:54,970 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=26, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:54,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:54,985 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:58:54,987 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:54,988 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390334987"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390334987"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390334987"}]},"ts":"1676390334987"} 2023-02-14 15:58:54,990 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:55,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:55,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:55,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:58:55,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:58:55,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:55,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:55,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:55,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:58:55,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:58:55,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:55,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:55,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:58:55,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:58:55,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:58:55,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:55,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:55,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:58:55,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:58:55,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:55,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:58:55,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:55,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:55,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:58:55,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:55,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:55,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:58:55,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:55,145 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:58:55,146 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390335145"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390335145"}]},"ts":"1676390335145"} 2023-02-14 15:58:55,147 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:58:55,148 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390335147"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390335147"}]},"ts":"1676390335147"} 2023-02-14 15:58:55,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:55,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:55,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:55,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:58:55,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:55,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:55,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:58:55,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:55,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:55,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:58:55,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:55,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:55,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:58:55,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:55,150 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:58:55,150 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390335150"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390335150"}]},"ts":"1676390335150"} 2023-02-14 15:58:55,159 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:58:55,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:55,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:58:55,161 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=26 2023-02-14 15:58:55,161 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 180 msec 2023-02-14 15:58:55,163 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:58:55,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-14 15:58:55,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 185 msec 2023-02-14 15:58:55,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:55,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:55,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:58:55,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:58:55,165 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:58:55,165 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 207 msec 2023-02-14 15:58:55,166 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390335165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390335165"}]},"ts":"1676390335165"} 2023-02-14 15:58:55,166 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-14 15:58:55,166 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 170 msec 2023-02-14 15:58:55,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:55,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:58:55,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:55,168 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 209 msec 2023-02-14 15:58:55,169 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:58:55,169 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390335169"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390335169"}]},"ts":"1676390335169"} 2023-02-14 15:58:55,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:55,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 212 msec 2023-02-14 15:58:55,171 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:58:55,171 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390335171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390335171"}]},"ts":"1676390335171"} 2023-02-14 15:58:55,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=23 2023-02-14 15:58:55,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 200 msec 2023-02-14 15:58:55,177 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 219 msec 2023-02-14 15:58:55,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-14 15:58:55,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 206 msec 2023-02-14 15:58:55,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 224 msec 2023-02-14 15:58:55,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=25 2023-02-14 15:58:55,187 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=25, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 216 msec 2023-02-14 15:58:55,190 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=22 2023-02-14 15:58:55,190 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 233 msec 2023-02-14 15:58:55,191 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390335190"}]},"ts":"1676390335190"} 2023-02-14 15:58:55,193 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:58:55,196 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:58:55,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-14 15:58:55,200 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 266 msec 2023-02-14 15:58:55,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-14 15:58:55,700 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 22 completed 2023-02-14 15:58:55,711 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:58:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390335711 (current time:1676390335711). 2023-02-14 15:58:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:58:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x66b79b39 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:55,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@305e857f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:58:55,724 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:58:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b79b39 to 127.0.0.1:55782 2023-02-14 15:58:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:58:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:58:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:58:55,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:58:55,746 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:58:55,747 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE; timeout=600000, timestamp=1676390935747 2023-02-14 15:58:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:58:55,747 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676390334053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 2023-02-14 15:58:55,753 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:58:55,753 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:58:55,775 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:58:55,785 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,785 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,788 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,791 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,791 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,796 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,797 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,797 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,797 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,797 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,797 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,798 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,799 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:55,799 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:55,806 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,807 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,807 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,807 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,808 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,811 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,813 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:58:55,928 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } as finished. 2023-02-14 15:58:55,928 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676390334053 2023-02-14 15:58:55,931 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390334053 2023-02-14 15:58:55,976 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390334053 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/emptySnaptb-1676390334053 2023-02-14 15:58:56,003 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:58:56,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:58:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' is still in progress! 2023-02-14 15:58:56,008 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-14 15:58:56,015 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676390334053 of table testRestoreSnapshot-1676390334053 completed 2023-02-14 15:58:56,015 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390334053 2023-02-14 15:58:56,015 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390334053 2023-02-14 15:58:56,015 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676390334053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 at 1676390335747Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } at 1676390335775 (+28 ms)Consolidate snapshot: emptySnaptb-1676390334053 at 1676390335928 (+153 ms)Loading Region manifests for emptySnaptb-1676390334053 at 1676390335928Writing data manifest for emptySnaptb-1676390334053 at 1676390335947 (+19 ms)Verifying snapshot: emptySnaptb-1676390334053 at 1676390335963 (+16 ms)Snapshot emptySnaptb-1676390334053 of table testRestoreSnapshot-1676390334053 completed at 1676390336014 (+51 ms) 2023-02-14 15:58:56,016 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:58:56,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE in 278 msec 2023-02-14 15:58:56,508 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:58:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:58:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:58:56,512 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676390334053 2023-02-14 15:58:56,517 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676390334053 2023-02-14 15:58:56,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:58:56,530 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676390334053 2023-02-14 15:58:56,531 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390336530"}]},"ts":"1676390336530"} 2023-02-14 15:58:56,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-14 15:58:56,533 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:58:56,537 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:58:56,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:58:56,543 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:58:56,543 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:58:56,544 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:58:56,544 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:58:56,545 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:56,545 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:56,545 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:58:56,545 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:58:56,546 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:56,545 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:58:56,546 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:58:56,547 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:56,547 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:58:56,547 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:58:56,548 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:56,548 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:58:56,550 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:56,550 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:58:56,696 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:58:56,696 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:56,696 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:56,696 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:56,696 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:56,696 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:56,697 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390336696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390336696"}]},"ts":"1676390336696"} 2023-02-14 15:58:56,697 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390336696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390336696"}]},"ts":"1676390336696"} 2023-02-14 15:58:56,697 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390336696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390336696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390336696"}]},"ts":"1676390336696"} 2023-02-14 15:58:56,697 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390336696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390336696"}]},"ts":"1676390336696"} 2023-02-14 15:58:56,697 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390336696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390336696"}]},"ts":"1676390336696"} 2023-02-14 15:58:56,700 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=39, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:56,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=41, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:56,703 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=42, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:56,705 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=40, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:56,706 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:56,707 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390336706"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390336706"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390336706"}]},"ts":"1676390336706"} 2023-02-14 15:58:56,707 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=38, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:56,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=37, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-14 15:58:56,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:56,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:56,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:58:56,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,861 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,861 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,862 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:58:56,862 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:58:56,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:56,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:58:56,863 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:58:56,863 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:58:56,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:56,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,864 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:58:56,865 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:58:56,865 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:56,866 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:56,869 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,872 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:58:56,872 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:58:56,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,872 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:58:56,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,874 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:56,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:56,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:56,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71991405, jitterRate=0.07275553047657013}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:56,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66675149, jitterRate=-0.006462857127189636}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:56,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:58:56,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:58:56,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=43, masterSystemTime=1676390336853 2023-02-14 15:58:56,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=44, masterSystemTime=1676390336854 2023-02-14 15:58:56,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:56,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:56,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:56,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:58:56,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:56,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,886 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:56,886 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336886"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390336886"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390336886"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390336886"}]},"ts":"1676390336886"} 2023-02-14 15:58:56,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:56,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:56,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:56,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:58:56,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,888 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:56,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:56,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,897 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336888"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390336888"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390336888"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390336888"}]},"ts":"1676390336888"} 2023-02-14 15:58:56,897 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=39 2023-02-14 15:58:56,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=39, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 190 msec 2023-02-14 15:58:56,900 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:58:56,900 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:58:56,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:56,900 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,901 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:58:56,902 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 361 msec 2023-02-14 15:58:56,902 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:58:56,903 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:58:56,903 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:56,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66248630, jitterRate=-0.012818485498428345}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:56,903 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:58:56,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:58:56,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=48, masterSystemTime=1676390336858 2023-02-14 15:58:56,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,910 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=41 2023-02-14 15:58:56,911 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=41, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 200 msec 2023-02-14 15:58:56,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:56,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:56,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:56,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:58:56,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:56,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,913 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:56,913 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390336913"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390336913"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390336913"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390336913"}]},"ts":"1676390336913"} 2023-02-14 15:58:56,914 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 374 msec 2023-02-14 15:58:56,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:56,915 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64102004, jitterRate=-0.044805705547332764}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:56,917 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:56,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:58:56,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=45, masterSystemTime=1676390336853 2023-02-14 15:58:56,918 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:58:56,918 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:58:56,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,919 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:58:56,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=37 2023-02-14 15:58:56,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=37, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 207 msec 2023-02-14 15:58:56,921 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:56,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:56,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:56,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,923 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:56,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,923 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390336923"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390336923"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390336923"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390336923"}]},"ts":"1676390336923"} 2023-02-14 15:58:56,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 383 msec 2023-02-14 15:58:56,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:56,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68190949, jitterRate=0.01612432301044464}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:56,931 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=42 2023-02-14 15:58:56,931 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=42, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 225 msec 2023-02-14 15:58:56,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:58:56,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=47, masterSystemTime=1676390336854 2023-02-14 15:58:56,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:56,934 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 394 msec 2023-02-14 15:58:56,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72430834, jitterRate=0.07930353283882141}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:56,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:58:56,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:56,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:56,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=46, masterSystemTime=1676390336858 2023-02-14 15:58:56,936 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:56,936 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336936"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390336936"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390336936"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390336936"}]},"ts":"1676390336936"} 2023-02-14 15:58:56,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:56,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:56,940 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:56,940 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390336939"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390336939"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390336939"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390336939"}]},"ts":"1676390336939"} 2023-02-14 15:58:56,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=38 2023-02-14 15:58:56,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=38, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 231 msec 2023-02-14 15:58:56,945 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 405 msec 2023-02-14 15:58:56,946 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=40 2023-02-14 15:58:56,947 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=40, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 237 msec 2023-02-14 15:58:56,950 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=36 2023-02-14 15:58:56,950 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 410 msec 2023-02-14 15:58:56,952 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390336952"}]},"ts":"1676390336952"} 2023-02-14 15:58:56,954 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:58:56,958 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676390334053' was successfully enabled. 2023-02-14 15:58:56,962 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676390334053 in 442 msec 2023-02-14 15:58:57,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-14 15:58:57,285 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 36 completed 2023-02-14 15:58:57,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:57,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:57,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:57,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:57,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:57,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:57,342 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676390334053 2023-02-14 15:58:57,343 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:57,344 INFO [Listener at localhost/39279] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 15:58:57,366 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:58:57,369 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:58:57,371 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:58:57,373 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:58:57,376 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:58:57,388 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:58:57,389 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:58:57,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:58:57,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-14 15:58:57,395 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390337395"}]},"ts":"1676390337395"} 2023-02-14 15:58:57,397 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:58:57,398 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:58:57,399 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:58:57,402 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:58:57,402 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:58:57,402 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:58:57,402 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:58:57,403 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:58:57,404 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:57,404 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:57,404 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337404"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390337404"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390337404"}]},"ts":"1676390337404"} 2023-02-14 15:58:57,404 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337404"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390337404"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390337404"}]},"ts":"1676390337404"} 2023-02-14 15:58:57,404 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:57,404 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:57,405 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337404"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390337404"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390337404"}]},"ts":"1676390337404"} 2023-02-14 15:58:57,405 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337404"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390337404"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390337404"}]},"ts":"1676390337404"} 2023-02-14 15:58:57,405 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:57,405 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390337405"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390337405"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390337405"}]},"ts":"1676390337405"} 2023-02-14 15:58:57,406 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=54, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:57,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=51, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:57,409 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=52, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:57,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=53, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:57,412 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:57,413 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:58:57,414 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:57,414 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390337414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390337414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390337414"}]},"ts":"1676390337414"} 2023-02-14 15:58:57,417 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=50, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:57,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:57,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:58:57,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:57,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:57,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:58:57,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:57,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:57,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:58:57,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:57,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:57,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:58:57,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:57,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 539ff23322a9c80991d27b3fbff0e4a6 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-14 15:58:57,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b3062a790bf6f13489f2b501c241d887 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-14 15:58:57,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:57,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:58:57,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:57,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:57,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:58:57,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:57,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing da0d94029320172fcd557222cbb7bf9d 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-14 15:58:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-14 15:58:57,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:58:57,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:58:57,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:58:57,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/.tmp/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:58:57,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/.tmp/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:58:57,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/.tmp/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:58:57,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/.tmp/cf/0ff98c593797444dbfe2d530ee4ea760 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:58:57,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/.tmp/cf/02b28f40d4864ccd9260cf45d273018c as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:58:57,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/.tmp/cf/d0cbb40341b14f699c493a7ab9a2b0b1 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:58:57,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760, entries=29, sequenceid=8, filesize=9.8 K 2023-02-14 15:58:57,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c, entries=36, sequenceid=8, filesize=11.0 K 2023-02-14 15:58:57,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1, entries=35, sequenceid=8, filesize=10.9 K 2023-02-14 15:58:57,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 539ff23322a9c80991d27b3fbff0e4a6 in 180ms, sequenceid=8, compaction requested=false 2023-02-14 15:58:57,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for b3062a790bf6f13489f2b501c241d887 in 180ms, sequenceid=8, compaction requested=false 2023-02-14 15:58:57,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for da0d94029320172fcd557222cbb7bf9d in 178ms, sequenceid=8, compaction requested=false 2023-02-14 15:58:57,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676390334053' 2023-02-14 15:58:57,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676390334053' 2023-02-14 15:58:57,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676390334053' 2023-02-14 15:58:57,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:58:57,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:58:57,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:58:57,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:57,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:58:57,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:57,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:58:57,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:57,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:58:57,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:57,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:57,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:58:57,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:57,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:57,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:58:57,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:57,759 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:58:57,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 70081217ed6e34352243aa2a9fed197b 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-14 15:58:57,759 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337759"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390337759"}]},"ts":"1676390337759"} 2023-02-14 15:58:57,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:57,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:57,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:58:57,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:57,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:57,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:58:57,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:57,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0e3b60cf882c8a008dc41bf06dfa1855 1/1 column families, dataSize=21.78 KB heapSize=47.08 KB 2023-02-14 15:58:57,762 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:58:57,762 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337762"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390337762"}]},"ts":"1676390337762"} 2023-02-14 15:58:57,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:57,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:57,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:58:57,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:57,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:57,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:58:57,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:57,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a703922cf39fc2176a548eb4e7bceb40 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-14 15:58:57,765 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:58:57,765 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390337765"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390337765"}]},"ts":"1676390337765"} 2023-02-14 15:58:57,774 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=54 2023-02-14 15:58:57,774 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=54, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 358 msec 2023-02-14 15:58:57,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=52 2023-02-14 15:58:57,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=52, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 363 msec 2023-02-14 15:58:57,777 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 375 msec 2023-02-14 15:58:57,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=50 2023-02-14 15:58:57,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=50, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 357 msec 2023-02-14 15:58:57,778 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 376 msec 2023-02-14 15:58:57,779 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 378 msec 2023-02-14 15:58:57,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:58:57,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:58:57,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/.tmp/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:58:57,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=21.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/.tmp/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:58:57,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/.tmp/cf/8385c17038f646799efbd15afd668ef2 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:58:57,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/.tmp/cf/6d30ae786a014cb4b9f248d9f924f34c as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:58:57,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2, entries=28, sequenceid=8, filesize=9.7 K 2023-02-14 15:58:57,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for a703922cf39fc2176a548eb4e7bceb40 in 87ms, sequenceid=8, compaction requested=false 2023-02-14 15:58:57,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c, entries=333, sequenceid=8, filesize=61.7 K 2023-02-14 15:58:57,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:58:57,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.78 KB/22307, heapSize ~47.06 KB/48192, currentSize=0 B/0 for 0e3b60cf882c8a008dc41bf06dfa1855 in 97ms, sequenceid=8, compaction requested=false 2023-02-14 15:58:57,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:57,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:58:57,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:57,862 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:58:57,862 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390337862"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390337862"}]},"ts":"1676390337862"} 2023-02-14 15:58:57,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:58:57,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:57,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:58:57,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:57,866 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:58:57,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=53 2023-02-14 15:58:57,866 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390337866"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390337866"}]},"ts":"1676390337866"} 2023-02-14 15:58:57,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=53, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 453 msec 2023-02-14 15:58:57,868 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 467 msec 2023-02-14 15:58:57,873 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-02-14 15:58:57,873 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 456 msec 2023-02-14 15:58:57,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 474 msec 2023-02-14 15:58:58,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-14 15:58:58,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:58:58,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/.tmp/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:58:58,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/.tmp/cf/237d1e590f534d84bfe7c745e4a9dca4 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:58:58,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4, entries=39, sequenceid=8, filesize=11.5 K 2023-02-14 15:58:58,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 70081217ed6e34352243aa2a9fed197b in 489ms, sequenceid=8, compaction requested=false 2023-02-14 15:58:58,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:58:58,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:58,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:58:58,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:58,257 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:58:58,257 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390338257"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390338257"}]},"ts":"1676390338257"} 2023-02-14 15:58:58,261 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=51 2023-02-14 15:58:58,261 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=51, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 851 msec 2023-02-14 15:58:58,263 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=49 2023-02-14 15:58:58,263 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 862 msec 2023-02-14 15:58:58,264 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390338264"}]},"ts":"1676390338264"} 2023-02-14 15:58:58,266 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:58:58,267 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:58:58,270 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 879 msec 2023-02-14 15:58:58,478 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 15:58:58,550 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-14 15:58:58,551 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-14 15:58:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-14 15:58:58,900 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 49 completed 2023-02-14 15:58:58,901 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:58:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390338901 (current time:1676390338901). 2023-02-14 15:58:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:58:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x69e3cb50 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:58:58,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5550df9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:58:58,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:58:58,911 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53476, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:58:58,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69e3cb50 to 127.0.0.1:55782 2023-02-14 15:58:58,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:58:58,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:58:58,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:58:58,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:58:58,918 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:58:58,919 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE; timeout=600000, timestamp=1676390938919 2023-02-14 15:58:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:58:58,919 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676390334053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 2023-02-14 15:58:58,919 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:58:58,919 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:58:58,944 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:58:58,949 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,949 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,950 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,950 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,960 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,960 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,961 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,961 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,961 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,967 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,968 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,968 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,971 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:58:58,971 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:58:58,974 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c] hfiles 2023-02-14 15:58:58,974 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:58:58,979 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1] hfiles 2023-02-14 15:58:58,979 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:58:58,980 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760] hfiles 2023-02-14 15:58:58,980 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4] hfiles 2023-02-14 15:58:58,981 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:58:58,981 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c] hfiles 2023-02-14 15:58:58,981 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:58:58,981 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:58:58,986 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc] hfiles 2023-02-14 15:58:58,986 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:58:58,987 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:58:58,987 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:58:58,987 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:58:58,987 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:58:58,987 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:58:58,991 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2] hfiles 2023-02-14 15:58:58,991 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:58:59,040 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } as finished. 2023-02-14 15:58:59,041 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676390334053 2023-02-14 15:58:59,042 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390334053 2023-02-14 15:58:59,089 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390334053 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/snaptb0-1676390334053 2023-02-14 15:58:59,137 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676390334053 of table testRestoreSnapshot-1676390334053 completed 2023-02-14 15:58:59,137 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390334053 2023-02-14 15:58:59,137 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390334053 2023-02-14 15:58:59,137 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676390334053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 at 1676390338919Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } at 1676390338944 (+25 ms)Consolidate snapshot: snaptb0-1676390334053 at 1676390339041 (+97 ms)Loading Region manifests for snaptb0-1676390334053 at 1676390339041Writing data manifest for snaptb0-1676390334053 at 1676390339057 (+16 ms)Verifying snapshot: snaptb0-1676390334053 at 1676390339073 (+16 ms)Snapshot snaptb0-1676390334053 of table testRestoreSnapshot-1676390334053 completed at 1676390339137 (+64 ms) 2023-02-14 15:58:59,139 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:58:59,141 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE in 223 msec 2023-02-14 15:58:59,170 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:58:59,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:58:59,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:58:59,172 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676390334053 2023-02-14 15:58:59,173 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676390334053 2023-02-14 15:58:59,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:58:59,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-14 15:58:59,178 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676390334053 2023-02-14 15:58:59,178 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390339178"}]},"ts":"1676390339178"} 2023-02-14 15:58:59,180 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:58:59,186 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:58:59,186 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:58:59,188 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:58:59,188 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:58:59,188 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:58:59,189 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:58:59,190 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:59,190 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:58:59,190 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:58:59,190 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:59,190 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:58:59,190 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:59,190 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:59,191 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:58:59,191 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:58:59,192 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:59,192 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:58:59,192 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:58:59,193 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:59,193 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:58:59,340 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:58:59,341 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:59,341 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:59,341 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:59,341 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:59,341 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:59,341 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390339341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390339341"}]},"ts":"1676390339341"} 2023-02-14 15:58:59,341 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390339341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390339341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390339341"}]},"ts":"1676390339341"} 2023-02-14 15:58:59,341 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390339341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390339341"}]},"ts":"1676390339341"} 2023-02-14 15:58:59,341 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390339341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390339341"}]},"ts":"1676390339341"} 2023-02-14 15:58:59,341 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390339341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390339341"}]},"ts":"1676390339341"} 2023-02-14 15:58:59,343 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=65, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:59,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=69, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:59,347 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=67, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:59,350 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=68, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:58:59,352 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=66, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:58:59,354 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:59,354 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390339354"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390339354"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390339354"}]},"ts":"1676390339354"} 2023-02-14 15:58:59,362 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=64, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:58:59,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-14 15:58:59,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:59,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:58:59,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:59,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:59,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:58:59,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,503 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:59,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,504 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:58:59,504 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:58:59,505 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:58:59,505 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,506 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:58:59,506 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:58:59,507 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:58:59,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:59,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:58:59,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:59,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,516 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,517 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:58:59,517 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:58:59,518 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:58:59,524 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:58:59,525 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:59,525 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:58:59,525 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:59,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,529 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:58:59,529 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:59,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:58:59,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:58:59,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70372529, jitterRate=0.04863239824771881}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:59,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:58:59,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=74, masterSystemTime=1676390339499 2023-02-14 15:58:59,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:58:59,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65033302, jitterRate=-0.03092828392982483}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:59,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:58:59,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59620381, jitterRate=-0.11158709228038788}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:59,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=73, masterSystemTime=1676390339497 2023-02-14 15:58:59,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:58:59,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=75, masterSystemTime=1676390339504 2023-02-14 15:58:59,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:59,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:58:59,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:59,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:58:59,546 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:59,546 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339545"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390339545"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390339545"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390339545"}]},"ts":"1676390339545"} 2023-02-14 15:58:59,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:59,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:59,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:58:59,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:59,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:58:59,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:59,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,550 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:59,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:59,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:59,550 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339550"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390339550"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390339550"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390339550"}]},"ts":"1676390339550"} 2023-02-14 15:58:59,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:59,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:58:59,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:58:59,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,551 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,552 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:59,552 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390339552"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390339552"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390339552"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390339552"}]},"ts":"1676390339552"} 2023-02-14 15:58:59,554 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:58:59,554 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:58:59,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=66 2023-02-14 15:58:59,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=66, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 198 msec 2023-02-14 15:58:59,556 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:58:59,560 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,562 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 369 msec 2023-02-14 15:58:59,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=68 2023-02-14 15:58:59,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=68, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 204 msec 2023-02-14 15:58:59,565 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=64 2023-02-14 15:58:59,565 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,565 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=64, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 197 msec 2023-02-14 15:58:59,566 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:58:59,566 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:58:59,566 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:58:59,567 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 378 msec 2023-02-14 15:58:59,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 379 msec 2023-02-14 15:58:59,568 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:58:59,568 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:58:59,568 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:58:59,573 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:58:59,576 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:59,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:58:59,586 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:58:59,586 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:59,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60972848, jitterRate=-0.09143376350402832}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:59,586 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:58:59,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:58:59,587 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:58:59,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=71, masterSystemTime=1676390339499 2023-02-14 15:58:59,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:59,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:58:59,591 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:58:59,591 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390339591"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390339591"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390339591"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390339591"}]},"ts":"1676390339591"} 2023-02-14 15:58:59,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:58:59,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:58:59,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69421311, jitterRate=0.03445814549922943}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:59,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:58:59,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62038982, jitterRate=-0.07554712891578674}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:58:59,597 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=69 2023-02-14 15:58:59,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:58:59,597 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=69, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 249 msec 2023-02-14 15:58:59,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=72, masterSystemTime=1676390339504 2023-02-14 15:58:59,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=70, masterSystemTime=1676390339497 2023-02-14 15:58:59,599 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 411 msec 2023-02-14 15:58:59,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:59,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:58:59,600 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:58:59,601 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339600"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390339600"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390339600"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390339600"}]},"ts":"1676390339600"} 2023-02-14 15:58:59,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:59,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:58:59,602 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:58:59,602 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390339602"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390339602"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390339602"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390339602"}]},"ts":"1676390339602"} 2023-02-14 15:58:59,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=67 2023-02-14 15:58:59,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=67, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 256 msec 2023-02-14 15:58:59,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 420 msec 2023-02-14 15:58:59,608 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=65 2023-02-14 15:58:59,608 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=65, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 262 msec 2023-02-14 15:58:59,610 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=63 2023-02-14 15:58:59,610 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 422 msec 2023-02-14 15:58:59,610 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390339610"}]},"ts":"1676390339610"} 2023-02-14 15:58:59,612 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:58:59,614 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676390334053' was successfully enabled. 2023-02-14 15:58:59,616 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676390334053 in 442 msec 2023-02-14 15:58:59,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-14 15:58:59,930 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 63 completed 2023-02-14 15:58:59,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:59,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:59,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:59,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:59,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:59,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:58:59,962 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676390334053 2023-02-14 15:58:59,962 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:58:59,964 INFO [Listener at localhost/39279] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 15:59:00,010 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-14 15:59:00,011 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-14 15:59:00,297 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:00,506 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:00,656 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:00,807 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:00,998 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:02,718 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(139): Connect 0x7042044c to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:02,728 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@700455dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:02,732 DEBUG [hconnection-0x6d06cc97-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:02,739 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53482, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:02,743 DEBUG [hconnection-0x6d06cc97-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:02,746 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:02,940 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:03,131 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:03,134 DEBUG [hconnection-0x6d06cc97-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:03,140 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46760, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:03,276 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:03,364 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 15:59:03,393 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:03,535 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:04,750 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:04,750 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:59:04,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:04,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-14 15:59:04,755 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390344755"}]},"ts":"1676390344755"} 2023-02-14 15:59:04,757 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:59:04,760 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:59:04,761 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:59:04,762 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:59:04,763 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:59:04,763 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:59:04,763 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:59:04,763 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:59:04,763 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:04,764 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390344763"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390344763"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390344763"}]},"ts":"1676390344763"} 2023-02-14 15:59:04,764 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:04,764 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:04,764 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390344764"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390344764"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390344764"}]},"ts":"1676390344764"} 2023-02-14 15:59:04,764 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390344764"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390344764"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390344764"}]},"ts":"1676390344764"} 2023-02-14 15:59:04,764 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:04,764 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390344764"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390344764"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390344764"}]},"ts":"1676390344764"} 2023-02-14 15:59:04,764 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:04,765 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390344764"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390344764"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390344764"}]},"ts":"1676390344764"} 2023-02-14 15:59:04,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=78, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:04,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=79, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:04,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=77, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:04,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=80, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:04,769 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:04,769 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:59:04,770 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:04,770 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390344770"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390344770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390344770"}]},"ts":"1676390344770"} 2023-02-14 15:59:04,772 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:04,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:59:04,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:59:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:04,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 70081217ed6e34352243aa2a9fed197b 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-14 15:59:04,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:04,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:59:04,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:04,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:59:04,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:04,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:04,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:59:04,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:04,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:04,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:59:04,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:04,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing da0d94029320172fcd557222cbb7bf9d 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-14 15:59:04,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b3062a790bf6f13489f2b501c241d887 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-14 15:59:04,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f 2023-02-14 15:59:04,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce 2023-02-14 15:59:04,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 2023-02-14 15:59:04,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/.tmp/cf/e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:04,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/.tmp/cf/9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:05,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/.tmp/cf/736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:05,001 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7042044c to 127.0.0.1:55782 2023-02-14 15:59:05,001 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:05,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/.tmp/cf/e6e75eef833c40099e6c444de9d5e253 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:05,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/.tmp/cf/9e9547d81f444025a05a320763442ca5 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:05,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-14 15:59:05,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253, entries=28, sequenceid=15, filesize=9.7 K 2023-02-14 15:59:05,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/.tmp/cf/736008cfbb8d47aea704b157afbae175 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:05,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5, entries=37, sequenceid=15, filesize=11.2 K 2023-02-14 15:59:05,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for da0d94029320172fcd557222cbb7bf9d in 85ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:05,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 70081217ed6e34352243aa2a9fed197b in 89ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:05,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175, entries=37, sequenceid=15, filesize=11.2 K 2023-02-14 15:59:05,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for b3062a790bf6f13489f2b501c241d887 in 94ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:05,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:05,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:05,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:05,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:05,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:05,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:05,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:05,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:05,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:05,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:05,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:59:05,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:59:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:05,029 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:59:05,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 539ff23322a9c80991d27b3fbff0e4a6 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-14 15:59:05,030 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390345029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390345029"}]},"ts":"1676390345029"} 2023-02-14 15:59:05,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:05,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:05,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:59:05,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:05,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:05,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:59:05,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:05,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a703922cf39fc2176a548eb4e7bceb40 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-14 15:59:05,031 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:59:05,031 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390345031"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390345031"}]},"ts":"1676390345031"} 2023-02-14 15:59:05,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:05,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:05,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:59:05,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:05,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:05,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:59:05,033 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:59:05,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:05,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0e3b60cf882c8a008dc41bf06dfa1855 1/1 column families, dataSize=21.78 KB heapSize=47.08 KB 2023-02-14 15:59:05,033 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390345033"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390345033"}]},"ts":"1676390345033"} 2023-02-14 15:59:05,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=78 2023-02-14 15:59:05,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=78, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 268 msec 2023-02-14 15:59:05,044 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=77 2023-02-14 15:59:05,044 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=77, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 267 msec 2023-02-14 15:59:05,045 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 281 msec 2023-02-14 15:59:05,045 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=79 2023-02-14 15:59:05,045 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=79, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 274 msec 2023-02-14 15:59:05,046 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 284 msec 2023-02-14 15:59:05,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 284 msec 2023-02-14 15:59:05,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e 2023-02-14 15:59:05,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c 2023-02-14 15:59:05,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e 2023-02-14 15:59:05,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/.tmp/cf/6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:05,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/.tmp/cf/6ddde8eb26e04634a1bc07e0a58b1159 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:05,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159, entries=39, sequenceid=15, filesize=11.5 K 2023-02-14 15:59:05,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for a703922cf39fc2176a548eb4e7bceb40 in 119ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:05,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/.tmp/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:05,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/.tmp/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:05,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b, entries=333, sequenceid=15, filesize=61.7 K 2023-02-14 15:59:05,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.78 KB/22307, heapSize ~47.06 KB/48192, currentSize=0 B/0 for 0e3b60cf882c8a008dc41bf06dfa1855 in 170ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:05,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:05,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:05,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:05,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:05,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:05,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:05,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:05,231 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:59:05,231 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390345231"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390345231"}]},"ts":"1676390345231"} 2023-02-14 15:59:05,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:05,232 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:59:05,232 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390345232"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390345232"}]},"ts":"1676390345232"} 2023-02-14 15:59:05,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=80 2023-02-14 15:59:05,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=80, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 466 msec 2023-02-14 15:59:05,236 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-14 15:59:05,236 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 465 msec 2023-02-14 15:59:05,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 474 msec 2023-02-14 15:59:05,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 475 msec 2023-02-14 15:59:05,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-14 15:59:05,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/.tmp/cf/a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:05,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/.tmp/cf/a84211fff4aa4e11b17352e147ba1969 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:05,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969, entries=26, sequenceid=15, filesize=9.3 K 2023-02-14 15:59:05,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 539ff23322a9c80991d27b3fbff0e4a6 in 513ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:05,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:05,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:05,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:05,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:05,556 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:59:05,556 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390345556"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390345556"}]},"ts":"1676390345556"} 2023-02-14 15:59:05,560 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-14 15:59:05,560 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 786 msec 2023-02-14 15:59:05,562 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=76 2023-02-14 15:59:05,562 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 799 msec 2023-02-14 15:59:05,563 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390345563"}]},"ts":"1676390345563"} 2023-02-14 15:59:05,565 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:59:05,567 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:59:05,569 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 817 msec 2023-02-14 15:59:06,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-14 15:59:06,259 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 76 completed 2023-02-14 15:59:06,260 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390346261 (current time:1676390346261). 2023-02-14 15:59:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:59:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x16c46f99 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:06,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@610a9f5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:06,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:06,271 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:06,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16c46f99 to 127.0.0.1:55782 2023-02-14 15:59:06,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:06,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:59:06,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:06,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:06,277 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:06,278 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE; timeout=600000, timestamp=1676390946278 2023-02-14 15:59:06,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:06,278 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676390334053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 2023-02-14 15:59:06,278 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:59:06,279 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:59:06,296 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:06,298 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,298 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,298 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,298 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,299 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,299 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,298 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,299 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,300 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,300 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,301 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,301 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,301 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,301 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5] hfiles 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253] hfiles 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:06,305 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969] hfiles 2023-02-14 15:59:06,305 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:06,305 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:06,305 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2] hfiles 2023-02-14 15:59:06,305 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:06,305 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc] hfiles 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c 2023-02-14 15:59:06,309 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b] hfiles 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175] hfiles 2023-02-14 15:59:06,312 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:06,304 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:06,312 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:06,312 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:06,312 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:06,345 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } as finished. 2023-02-14 15:59:06,345 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676390334053 2023-02-14 15:59:06,346 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb1-1676390334053 2023-02-14 15:59:06,386 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb1-1676390334053 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/snaptb1-1676390334053 2023-02-14 15:59:06,420 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676390334053 of table testRestoreSnapshot-1676390334053 completed 2023-02-14 15:59:06,420 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb1-1676390334053 2023-02-14 15:59:06,421 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb1-1676390334053 2023-02-14 15:59:06,421 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676390334053 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 at 1676390346278Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } at 1676390346296 (+18 ms)Consolidate snapshot: snaptb1-1676390334053 at 1676390346345 (+49 ms)Loading Region manifests for snaptb1-1676390334053 at 1676390346345Writing data manifest for snaptb1-1676390334053 at 1676390346353 (+8 ms)Verifying snapshot: snaptb1-1676390334053 at 1676390346363 (+10 ms)Snapshot snaptb1-1676390334053 of table testRestoreSnapshot-1676390334053 completed at 1676390346420 (+57 ms) 2023-02-14 15:59:06,422 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:06,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE in 147 msec 2023-02-14 15:59:06,529 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:59:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:59:06,547 INFO [Listener at localhost/39279] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:06,548 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:06,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390346548 (current time:1676390346548). 2023-02-14 15:59:06,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:59:06,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x79516071 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@676d6eba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:06,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:06,559 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45520, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79516071 to 127.0.0.1:55782 2023-02-14 15:59:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:59:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:06,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:06,564 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:06,564 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE; timeout=600000, timestamp=1676390946564 2023-02-14 15:59:06,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:06,564 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676390334053-1676390346547 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 2023-02-14 15:59:06,565 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:59:06,565 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:59:06,582 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:06,585 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,585 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,588 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,588 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,588 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,588 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,588 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,589 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,590 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,590 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,590 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,590 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969] hfiles 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b] hfiles 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253] hfiles 2023-02-14 15:59:06,590 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175] hfiles 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:06,593 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2] hfiles 2023-02-14 15:59:06,593 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:06,593 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:06,592 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:06,593 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:06,593 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:06,593 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:06,594 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:06,594 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:06,607 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5] hfiles 2023-02-14 15:59:06,607 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:06,607 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc] hfiles 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c 2023-02-14 15:59:06,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:59:06,667 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } as finished. 2023-02-14 15:59:06,667 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:06,667 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:06,706 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676390334053-1676390346547 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:06,729 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676390334053-1676390346547 of table testRestoreSnapshot-1676390334053 completed 2023-02-14 15:59:06,729 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:06,729 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:06,729 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676390334053-1676390346547 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 at 1676390346565Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } at 1676390346582 (+17 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676390334053-1676390346547 at 1676390346667 (+85 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676390334053-1676390346547 at 1676390346667Writing data manifest for hbase-failsafe-snaptb0-1676390334053-1676390346547 at 1676390346673 (+6 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676390334053-1676390346547 at 1676390346689 (+16 ms)Snapshot hbase-failsafe-snaptb0-1676390334053-1676390346547 of table testRestoreSnapshot-1676390334053 completed at 1676390346729 (+40 ms) 2023-02-14 15:59:06,732 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:06,733 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE in 170 msec 2023-02-14 15:59:06,815 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:06,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:59:06,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676390334053-1676390346547 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:59:06,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "snaptb0-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390338901 type: DISABLED version: 2 ttl: 0 ) 2023-02-14 15:59:06,854 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676390334053 as table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:06,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-14 15:59:06,883 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000002 2023-02-14 15:59:06,884 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000001 2023-02-14 15:59:06,884 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000002 2023-02-14 15:59:06,886 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:06,891 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390338901 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:06,892 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053 2023-02-14 15:59:06,901 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:06,911 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e 2023-02-14 15:59:06,913 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce 2023-02-14 15:59:06,915 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 2023-02-14 15:59:06,916 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e 2023-02-14 15:59:06,918 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f 2023-02-14 15:59:06,919 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c 2023-02-14 15:59:06,919 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:06,919 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:06,919 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:06,919 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:06,919 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:06,919 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:06,950 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:06,951 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:06,952 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:06,952 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:06,955 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:06,956 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:06,956 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390338901 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:06,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390346958"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:06,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390346958"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:06,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390346958"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:06,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390346958"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:06,959 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390346958"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:06,959 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390346958"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:06,962 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-14 15:59:06,962 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}] 2023-02-14 15:59:06,963 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390346959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390346959"}]},"ts":"1676390346959"} 2023-02-14 15:59:06,963 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390346959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390346959"}]},"ts":"1676390346959"} 2023-02-14 15:59:06,963 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390346959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390346959"}]},"ts":"1676390346959"} 2023-02-14 15:59:06,963 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390346959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390346959"}]},"ts":"1676390346959"} 2023-02-14 15:59:06,963 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390346959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390346959"}]},"ts":"1676390346959"} 2023-02-14 15:59:06,963 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390346959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390346959"}]},"ts":"1676390346959"} 2023-02-14 15:59:06,968 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-14 15:59:06,968 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-14 15:59:06,969 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}] 2023-02-14 15:59:06,969 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676390334053 completed! 2023-02-14 15:59:06,972 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "snaptb0-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390338901 type: DISABLED version: 2 ttl: 0 ) in 129 msec 2023-02-14 15:59:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-14 15:59:07,110 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676390334053, procId: 91 completed 2023-02-14 15:59:07,111 INFO [Listener at localhost/39279] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:07,115 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676390334053-1676390346547" 2023-02-14 15:59:07,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676390334053-1676390346547 2023-02-14 15:59:07,120 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676390334053 2023-02-14 15:59:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:07,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-14 15:59:07,134 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676390334053 2023-02-14 15:59:07,134 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390347134"}]},"ts":"1676390347134"} 2023-02-14 15:59:07,136 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:59:07,140 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:59:07,143 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:59:07,145 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:59:07,145 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:59:07,146 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:59:07,146 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:59:07,147 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:59:07,147 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:07,147 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:07,147 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:07,147 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:07,148 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:07,148 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:59:07,149 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:07,297 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:59:07,298 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:59:07,298 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:59:07,298 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:59:07,298 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:59:07,302 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:07,302 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:07,302 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:07,302 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:07,302 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390347302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390347302"}]},"ts":"1676390347302"} 2023-02-14 15:59:07,302 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390347302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390347302"}]},"ts":"1676390347302"} 2023-02-14 15:59:07,302 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390347302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390347302"}]},"ts":"1676390347302"} 2023-02-14 15:59:07,302 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390347302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390347302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390347302"}]},"ts":"1676390347302"} 2023-02-14 15:59:07,302 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:07,303 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390347302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390347302"}]},"ts":"1676390347302"} 2023-02-14 15:59:07,309 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=96, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:07,310 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=95, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:07,312 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:07,313 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390347312"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390347312"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390347312"}]},"ts":"1676390347312"} 2023-02-14 15:59:07,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=98, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:07,317 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=97, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:07,319 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=94, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:07,320 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=93, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:07,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-14 15:59:07,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:07,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:07,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:07,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,472 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,472 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:07,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:07,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:07,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,475 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:07,475 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:07,475 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:59:07,478 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,479 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:07,479 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:07,480 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:59:07,483 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:07,487 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:07,487 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:07,490 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:59:07,490 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:07,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:07,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60715756, jitterRate=-0.095264732837677}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:07,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:07,500 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:07,500 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:07,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=100, masterSystemTime=1676390347462 2023-02-14 15:59:07,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,504 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:07,504 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:07,506 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:07,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:07,506 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347506"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390347506"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390347506"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390347506"}]},"ts":"1676390347506"} 2023-02-14 15:59:07,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:07,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:07,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:07,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:07,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=95 2023-02-14 15:59:07,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=95, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 199 msec 2023-02-14 15:59:07,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 370 msec 2023-02-14 15:59:07,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:07,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66808503, jitterRate=-0.004475727677345276}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:07,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:07,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=99, masterSystemTime=1676390347461 2023-02-14 15:59:07,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:07,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:07,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:07,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:07,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65720216, jitterRate=-0.02069246768951416}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:07,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:07,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:07,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:07,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,523 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:07,524 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347523"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390347523"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390347523"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390347523"}]},"ts":"1676390347523"} 2023-02-14 15:59:07,524 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=103, masterSystemTime=1676390347470 2023-02-14 15:59:07,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:07,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:07,526 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:07,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:07,527 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347526"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390347526"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390347526"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390347526"}]},"ts":"1676390347526"} 2023-02-14 15:59:07,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:07,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:07,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,529 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=96 2023-02-14 15:59:07,529 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=96, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 217 msec 2023-02-14 15:59:07,531 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 386 msec 2023-02-14 15:59:07,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=94 2023-02-14 15:59:07,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=94, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 210 msec 2023-02-14 15:59:07,532 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,533 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:07,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 392 msec 2023-02-14 15:59:07,533 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:07,534 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:59:07,536 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,540 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:07,540 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:07,541 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:59:07,545 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:07,545 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:07,545 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:59:07,555 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:07,555 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:07,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:07,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58961252, jitterRate=-0.12140887975692749}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:07,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:07,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=101, masterSystemTime=1676390347462 2023-02-14 15:59:07,565 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:07,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:07,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:07,567 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:07,567 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390347567"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390347567"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390347567"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390347567"}]},"ts":"1676390347567"} 2023-02-14 15:59:07,569 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=98 2023-02-14 15:59:07,570 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=98, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 254 msec 2023-02-14 15:59:07,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 427 msec 2023-02-14 15:59:07,572 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:07,573 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:07,573 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:07,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:07,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:07,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61513588, jitterRate=-0.08337610960006714}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:07,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:07,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64391061, jitterRate=-0.04049842059612274}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:07,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:07,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=102, masterSystemTime=1676390347470 2023-02-14 15:59:07,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=104, masterSystemTime=1676390347461 2023-02-14 15:59:07,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:07,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:07,580 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:07,580 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390347580"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390347580"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390347580"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390347580"}]},"ts":"1676390347580"} 2023-02-14 15:59:07,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:07,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:07,584 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:07,585 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390347584"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390347584"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390347584"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390347584"}]},"ts":"1676390347584"} 2023-02-14 15:59:07,587 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=97 2023-02-14 15:59:07,587 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=97, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 268 msec 2023-02-14 15:59:07,589 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 444 msec 2023-02-14 15:59:07,589 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=93 2023-02-14 15:59:07,589 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=93, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 266 msec 2023-02-14 15:59:07,591 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=92 2023-02-14 15:59:07,591 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 449 msec 2023-02-14 15:59:07,592 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390347592"}]},"ts":"1676390347592"} 2023-02-14 15:59:07,593 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:59:07,595 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676390334053' was successfully enabled. 2023-02-14 15:59:07,597 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676390334053 in 475 msec 2023-02-14 15:59:07,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-14 15:59:07,884 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 92 completed 2023-02-14 15:59:07,884 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(139): Connect 0x1b384ceb to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:07,890 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@aebe9d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:07,892 DEBUG [hconnection-0x272618c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:07,896 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:07,900 DEBUG [hconnection-0x272618c8-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:07,902 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36166, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:08,020 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:08,147 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:08,151 DEBUG [hconnection-0x272618c8-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:08,154 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45048, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:08,240 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:08,320 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:08,420 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:09,489 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:09,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:59:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:09,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-14 15:59:09,493 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390349493"}]},"ts":"1676390349493"} 2023-02-14 15:59:09,494 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:59:09,497 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:59:09,497 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:59:09,499 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:59:09,499 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:59:09,499 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:59:09,499 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:59:09,499 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:59:09,499 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:09,500 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349499"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390349499"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390349499"}]},"ts":"1676390349499"} 2023-02-14 15:59:09,500 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:09,500 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:09,500 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390349500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390349500"}]},"ts":"1676390349500"} 2023-02-14 15:59:09,500 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390349500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390349500"}]},"ts":"1676390349500"} 2023-02-14 15:59:09,500 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:09,500 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:09,500 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390349500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390349500"}]},"ts":"1676390349500"} 2023-02-14 15:59:09,500 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390349500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390349500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390349500"}]},"ts":"1676390349500"} 2023-02-14 15:59:09,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=110, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:09,504 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=108, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:09,505 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=109, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:09,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=107, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:09,506 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:59:09,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=111, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:09,507 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:09,507 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390349507"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390349507"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390349507"}]},"ts":"1676390349507"} 2023-02-14 15:59:09,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=106, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:09,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:09,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:09,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:59:09,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:09,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:09,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:09,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:59:09,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:09,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-14 15:59:09,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-14 15:59:09,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:09,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:09,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:09,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:09,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:09,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:09,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:59:09,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:09,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:09,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:59:09,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:09,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-14 15:59:09,672 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:59:09,672 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349672"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390349672"}]},"ts":"1676390349672"} 2023-02-14 15:59:09,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:09,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:09,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:59:09,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:09,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:09,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:59:09,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:09,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:09,674 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:09,674 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:59:09,674 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349674"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390349674"}]},"ts":"1676390349674"} 2023-02-14 15:59:09,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:09,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:09,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:59:09,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:09,675 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:59:09,675 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390349675"}]},"ts":"1676390349675"} 2023-02-14 15:59:09,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:09,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:59:09,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:09,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=110 2023-02-14 15:59:09,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=110, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 173 msec 2023-02-14 15:59:09,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-14 15:59:09,679 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=109 2023-02-14 15:59:09,679 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=109, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 171 msec 2023-02-14 15:59:09,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:09,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:09,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-14 15:59:09,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:09,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:09,682 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 181 msec 2023-02-14 15:59:09,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=108 2023-02-14 15:59:09,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=108, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 175 msec 2023-02-14 15:59:09,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-14 15:59:09,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:09,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 182 msec 2023-02-14 15:59:09,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:09,683 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:59:09,683 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390349683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390349683"}]},"ts":"1676390349683"} 2023-02-14 15:59:09,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:09,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:09,684 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:59:09,684 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390349684"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390349684"}]},"ts":"1676390349684"} 2023-02-14 15:59:09,684 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 185 msec 2023-02-14 15:59:09,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:09,686 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:59:09,686 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390349686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390349686"}]},"ts":"1676390349686"} 2023-02-14 15:59:09,686 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=106 2023-02-14 15:59:09,686 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=106, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 176 msec 2023-02-14 15:59:09,688 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=107 2023-02-14 15:59:09,688 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=107, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 179 msec 2023-02-14 15:59:09,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 189 msec 2023-02-14 15:59:09,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=111 2023-02-14 15:59:09,689 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 191 msec 2023-02-14 15:59:09,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=111, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 180 msec 2023-02-14 15:59:09,691 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=105 2023-02-14 15:59:09,691 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 192 msec 2023-02-14 15:59:09,691 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390349691"}]},"ts":"1676390349691"} 2023-02-14 15:59:09,692 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:59:09,694 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:59:09,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 205 msec 2023-02-14 15:59:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-14 15:59:09,744 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 105 completed 2023-02-14 15:59:09,752 INFO [Listener at localhost/39279] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:09,753 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390349753 (current time:1676390349753). 2023-02-14 15:59:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:59:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x5f03ddfa to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8f4594a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:09,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:09,761 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45542, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f03ddfa to 127.0.0.1:55782 2023-02-14 15:59:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:59:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:09,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:09,765 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:09,765 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE; timeout=600000, timestamp=1676390949765 2023-02-14 15:59:09,765 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676390334053-1676390349752 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 2023-02-14 15:59:09,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:09,766 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:59:09,766 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:59:09,774 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:09,776 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,776 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,776 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,776 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,776 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,776 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,777 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,777 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,777 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,777 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,777 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,777 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,779 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1] hfiles 2023-02-14 15:59:09,779 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:09,780 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2] hfiles 2023-02-14 15:59:09,780 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c] hfiles 2023-02-14 15:59:09,780 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:09,780 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:09,781 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:09,781 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:09,781 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c] hfiles 2023-02-14 15:59:09,781 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:09,782 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760] hfiles 2023-02-14 15:59:09,782 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc] hfiles 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:59:09,789 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:59:09,790 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4] hfiles 2023-02-14 15:59:09,790 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:09,831 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } as finished. 2023-02-14 15:59:09,831 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:09,831 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:09,852 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676390334053-1676390349752 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:09,871 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676390334053-1676390349752 of table testRestoreSnapshot-1676390334053 completed 2023-02-14 15:59:09,871 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:09,872 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:09,872 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676390334053-1676390349752 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 at 1676390349765Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } at 1676390349774 (+9 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676390334053-1676390349752 at 1676390349831 (+57 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676390334053-1676390349752 at 1676390349831Writing data manifest for hbase-failsafe-emptySnaptb-1676390334053-1676390349752 at 1676390349834 (+3 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676390334053-1676390349752 at 1676390349844 (+10 ms)Snapshot hbase-failsafe-emptySnaptb-1676390334053-1676390349752 of table testRestoreSnapshot-1676390334053 completed at 1676390349871 (+27 ms) 2023-02-14 15:59:09,873 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:09,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE in 110 msec 2023-02-14 15:59:10,016 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:10,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:59:10,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676390334053-1676390349752 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:59:10,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "emptySnaptb-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390335711 type: DISABLED version: 2 ttl: 0 ) 2023-02-14 15:59:10,024 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676390334053 as table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-14 15:59:10,042 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000003 2023-02-14 15:59:10,043 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000002 2023-02-14 15:59:10,043 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000003 2023-02-14 15:59:10,044 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:10,045 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390335711 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:10,045 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,053 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,056 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:59:10,057 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:59:10,059 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:59:10,060 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:59:10,061 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:59:10,062 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:59:10,063 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,063 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,063 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,063 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,063 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,063 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,068 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:10,069 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:10,070 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:10,070 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:10,070 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:10,071 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:10,072 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390335711 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:10,073 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390350073"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:10,073 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390350073"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:10,073 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390350073"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:10,073 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390350073"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:10,073 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390350073"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:10,073 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390350073"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:10,076 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}] 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390350074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390350074"}]},"ts":"1676390350074"} 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390350074"}]},"ts":"1676390350074"} 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390350074"}]},"ts":"1676390350074"} 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390350074"}]},"ts":"1676390350074"} 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390350074"}]},"ts":"1676390350074"} 2023-02-14 15:59:10,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390350074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390350074"}]},"ts":"1676390350074"} 2023-02-14 15:59:10,078 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-14 15:59:10,078 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-14 15:59:10,078 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}] 2023-02-14 15:59:10,079 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676390334053 completed! 2023-02-14 15:59:10,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "emptySnaptb-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390335711 type: DISABLED version: 2 ttl: 0 ) in 56 msec 2023-02-14 15:59:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-14 15:59:10,278 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676390334053, procId: 119 completed 2023-02-14 15:59:10,278 INFO [Listener at localhost/39279] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:10,279 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676390334053-1676390349752" 2023-02-14 15:59:10,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676390334053-1676390349752 2023-02-14 15:59:10,282 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,283 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-14 15:59:10,299 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676390334053 2023-02-14 15:59:10,299 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390350299"}]},"ts":"1676390350299"} 2023-02-14 15:59:10,300 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:59:10,303 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:59:10,303 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:59:10,305 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:59:10,305 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:59:10,305 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:59:10,305 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:59:10,306 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:10,306 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:10,306 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:10,306 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:10,308 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:59:10,308 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:10,312 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:59:10,313 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:10,456 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:59:10,457 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:59:10,457 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:59:10,457 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:59:10,457 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:59:10,461 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:10,461 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:10,461 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:10,462 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390350461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390350461"}]},"ts":"1676390350461"} 2023-02-14 15:59:10,462 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390350461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390350461"}]},"ts":"1676390350461"} 2023-02-14 15:59:10,461 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:10,461 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:10,462 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390350461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390350461"}]},"ts":"1676390350461"} 2023-02-14 15:59:10,462 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390350461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390350461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390350461"}]},"ts":"1676390350461"} 2023-02-14 15:59:10,462 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390350461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390350461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390350461"}]},"ts":"1676390350461"} 2023-02-14 15:59:10,463 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=124, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:10,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=125, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:10,465 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=123, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:10,466 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=121, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:10,467 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:10,467 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390350467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390350467"}]},"ts":"1676390350467"} 2023-02-14 15:59:10,467 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=126, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:10,468 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=122, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:10,528 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 15:59:10,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-14 15:59:10,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:10,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:10,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:10,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:10,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:10,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,622 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,622 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,622 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,624 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:10,624 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:10,624 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:10,624 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:10,624 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:59:10,624 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:59:10,625 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:10,625 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:10,625 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:59:10,626 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:10,626 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:10,626 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:10,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:10,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:10,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:10,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64975234, jitterRate=-0.031793564558029175}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:10,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62301316, jitterRate=-0.07163804769515991}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:10,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:10,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:10,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=129, masterSystemTime=1676390350617 2023-02-14 15:59:10,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=128, masterSystemTime=1676390350616 2023-02-14 15:59:10,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59023552, jitterRate=-0.12048053741455078}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:10,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:10,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:10,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:10,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:10,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=127, masterSystemTime=1676390350616 2023-02-14 15:59:10,637 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:10,637 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350637"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390350637"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390350637"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390350637"}]},"ts":"1676390350637"} 2023-02-14 15:59:10,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:10,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:10,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:10,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:10,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:10,639 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:10,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:10,639 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350639"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390350639"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390350639"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390350639"}]},"ts":"1676390350639"} 2023-02-14 15:59:10,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,639 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:10,640 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350639"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390350639"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390350639"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390350639"}]},"ts":"1676390350639"} 2023-02-14 15:59:10,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:10,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:10,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:10,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:10,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:10,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:10,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,643 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,643 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,644 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=123 2023-02-14 15:59:10,644 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=123, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,40531,1676390331113 in 175 msec 2023-02-14 15:59:10,645 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=125 2023-02-14 15:59:10,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=125, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,37877,1676390331146 in 178 msec 2023-02-14 15:59:10,646 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:10,646 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:10,646 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:10,647 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:10,646 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 341 msec 2023-02-14 15:59:10,646 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=124 2023-02-14 15:59:10,647 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=124, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,43707,1676390331069 in 180 msec 2023-02-14 15:59:10,647 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:59:10,647 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 343 msec 2023-02-14 15:59:10,647 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:10,647 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:10,647 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:59:10,648 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:59:10,648 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 343 msec 2023-02-14 15:59:10,648 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:10,648 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:10,649 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:10,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:10,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:10,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:10,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73442685, jitterRate=0.09438128769397736}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:10,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:10,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68765381, jitterRate=0.024684026837348938}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:10,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:10,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62035943, jitterRate=-0.07559241354465485}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:10,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:10,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=132, masterSystemTime=1676390350616 2023-02-14 15:59:10,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=130, masterSystemTime=1676390350616 2023-02-14 15:59:10,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=131, masterSystemTime=1676390350617 2023-02-14 15:59:10,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:10,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:10,657 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:10,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390350657"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390350657"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390350657"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390350657"}]},"ts":"1676390350657"} 2023-02-14 15:59:10,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:10,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:10,658 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:10,658 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390350658"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390350658"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390350658"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390350658"}]},"ts":"1676390350658"} 2023-02-14 15:59:10,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:10,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:10,660 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:10,660 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390350659"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390350659"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390350659"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390350659"}]},"ts":"1676390350659"} 2023-02-14 15:59:10,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=122 2023-02-14 15:59:10,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=122, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146 in 191 msec 2023-02-14 15:59:10,664 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=126 2023-02-14 15:59:10,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 359 msec 2023-02-14 15:59:10,664 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=126, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113 in 194 msec 2023-02-14 15:59:10,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=121 2023-02-14 15:59:10,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=121, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,43707,1676390331069 in 196 msec 2023-02-14 15:59:10,665 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 361 msec 2023-02-14 15:59:10,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=121, resume processing ppid=120 2023-02-14 15:59:10,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 362 msec 2023-02-14 15:59:10,667 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390350667"}]},"ts":"1676390350667"} 2023-02-14 15:59:10,668 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:59:10,670 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676390334053' was successfully enabled. 2023-02-14 15:59:10,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676390334053 in 387 msec 2023-02-14 15:59:11,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-14 15:59:11,051 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 120 completed 2023-02-14 15:59:11,052 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(139): Connect 0x2c2934c9 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:11,058 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f3a7cee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:11,059 DEBUG [hconnection-0x43ec0260-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:11,062 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45552, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:11,070 DEBUG [hconnection-0x43ec0260-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:11,074 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:11,075 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:11,078 DEBUG [hconnection-0x43ec0260-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:11,080 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36178, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:11,081 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:11,085 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:11,088 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:11,091 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:11,097 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:11,097 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:59:11,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:11,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-14 15:59:11,105 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390351105"}]},"ts":"1676390351105"} 2023-02-14 15:59:11,107 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:59:11,110 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:59:11,112 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:59:11,114 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:59:11,114 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:59:11,114 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:59:11,114 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:59:11,114 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:59:11,115 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:11,115 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390351115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390351115"}]},"ts":"1676390351115"} 2023-02-14 15:59:11,115 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:11,115 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:11,116 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390351115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390351115"}]},"ts":"1676390351115"} 2023-02-14 15:59:11,115 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:11,115 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:11,116 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390351115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390351115"}]},"ts":"1676390351115"} 2023-02-14 15:59:11,116 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390351115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390351115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390351115"}]},"ts":"1676390351115"} 2023-02-14 15:59:11,116 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390351115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390351115"}]},"ts":"1676390351115"} 2023-02-14 15:59:11,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=138, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:11,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=136, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:11,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=135, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:11,120 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:59:11,121 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=139, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:11,121 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:11,121 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390351121"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390351121"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390351121"}]},"ts":"1676390351121"} 2023-02-14 15:59:11,122 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=137, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:11,122 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:11,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:11,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:11,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:59:11,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:59:11,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:11,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:11,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:11,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:11,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:59:11,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:59:11,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:11,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:11,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:11,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:59:11,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:11,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:11,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:59:11,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:11,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-14 15:59:11,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:11,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:11,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-14 15:59:11,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:11,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:11,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:11,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:11,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:59:11,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:11,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:11,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:59:11,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:11,281 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:59:11,281 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390351281"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390351281"}]},"ts":"1676390351281"} 2023-02-14 15:59:11,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-14 15:59:11,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:11,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:11,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:11,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:59:11,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:11,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:11,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:59:11,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:11,283 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:59:11,283 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351283"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390351283"}]},"ts":"1676390351283"} 2023-02-14 15:59:11,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:11,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:11,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:59:11,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:11,286 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:59:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:59:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:11,286 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390351286"}]},"ts":"1676390351286"} 2023-02-14 15:59:11,286 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=139 2023-02-14 15:59:11,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=139, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113 in 162 msec 2023-02-14 15:59:11,288 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 174 msec 2023-02-14 15:59:11,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=137 2023-02-14 15:59:11,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=137, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,43707,1676390331069 in 166 msec 2023-02-14 15:59:11,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-14 15:59:11,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-14 15:59:11,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:11,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:11,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=135 2023-02-14 15:59:11,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=135, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146 in 171 msec 2023-02-14 15:59:11,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 178 msec 2023-02-14 15:59:11,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:11,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:11,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 179 msec 2023-02-14 15:59:11,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:11,293 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:59:11,293 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351293"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390351293"}]},"ts":"1676390351293"} 2023-02-14 15:59:11,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:11,293 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:59:11,294 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390351293"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390351293"}]},"ts":"1676390351293"} 2023-02-14 15:59:11,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-14 15:59:11,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:11,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:11,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=138 2023-02-14 15:59:11,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=138, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,37877,1676390331146 in 178 msec 2023-02-14 15:59:11,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:11,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-02-14 15:59:11,303 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:59:11,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,43707,1676390331069 in 174 msec 2023-02-14 15:59:11,303 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390351303"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390351303"}]},"ts":"1676390351303"} 2023-02-14 15:59:11,304 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 190 msec 2023-02-14 15:59:11,305 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 191 msec 2023-02-14 15:59:11,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=136 2023-02-14 15:59:11,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=136, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,40531,1676390331113 in 188 msec 2023-02-14 15:59:11,308 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=133 2023-02-14 15:59:11,308 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 194 msec 2023-02-14 15:59:11,310 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390351309"}]},"ts":"1676390351309"} 2023-02-14 15:59:11,311 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:59:11,312 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:59:11,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 216 msec 2023-02-14 15:59:11,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-14 15:59:11,352 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 133 completed 2023-02-14 15:59:11,363 INFO [Listener at localhost/39279] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:11,364 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:11,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390351364 (current time:1676390351364). 2023-02-14 15:59:11,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:59:11,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x6763cf20 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:11,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@669ddf2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:11,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:11,373 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45562, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6763cf20 to 127.0.0.1:55782 2023-02-14 15:59:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:11,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:59:11,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:11,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:11,377 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:11,378 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE; timeout=600000, timestamp=1676390951378 2023-02-14 15:59:11,378 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676390334053-1676390351362 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 2023-02-14 15:59:11,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } 2023-02-14 15:59:11,379 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:59:11,379 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:59:11,390 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:11,403 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,403 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,403 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,403 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,405 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,406 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,406 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:11,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:11,406 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,409 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,409 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,410 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,412 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:11,629 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:11,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:59:11,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' is still in progress! 2023-02-14 15:59:11,630 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-14 15:59:11,890 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } as finished. 2023-02-14 15:59:11,890 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:11,891 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:11,915 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676390334053-1676390351362 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:11,954 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676390334053-1676390351362 of table testRestoreSnapshot-1676390334053 completed 2023-02-14 15:59:11,955 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:11,955 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:11,955 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676390334053-1676390351362 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676390334053 at 1676390351378Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } at 1676390351390 (+12 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676390334053-1676390351362 at 1676390351890 (+500 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676390334053-1676390351362 at 1676390351890Writing data manifest for hbase-failsafe-snaptb1-1676390334053-1676390351362 at 1676390351895 (+5 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676390334053-1676390351362 at 1676390351906 (+11 ms)Snapshot hbase-failsafe-snaptb1-1676390334053-1676390351362 of table testRestoreSnapshot-1676390334053 completed at 1676390351954 (+48 ms) 2023-02-14 15:59:11,957 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE 2023-02-14 15:59:11,959 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676390334053, type=EXCLUSIVE in 581 msec 2023-02-14 15:59:12,131 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:12,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 } is done 2023-02-14 15:59:12,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676390334053-1676390351362 table=testRestoreSnapshot-1676390334053 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:59:12,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 ) 2023-02-14 15:59:12,150 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676390334053 as table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-14 15:59:12,162 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000004 2023-02-14 15:59:12,163 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000003 2023-02-14 15:59:12,163 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000004 2023-02-14 15:59:12,164 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } 2023-02-14 15:59:12,166 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:12,166 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,173 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,174 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,177 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,181 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,185 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,187 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,191 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,194 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,197 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,201 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,204 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,208 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,211 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc of snapshot=snaptb1-1676390334053 to region=8852f07d6ff59754d6b918836384d34f table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,215 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,215 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,215 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,215 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,215 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,215 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,217 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 6d30ae786a014cb4b9f248d9f924f34c of snapshot=snaptb1-1676390334053 to region=0e3b60cf882c8a008dc41bf06dfa1855 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,217 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d0cbb40341b14f699c493a7ab9a2b0b1 of snapshot=snaptb1-1676390334053 to region=da0d94029320172fcd557222cbb7bf9d table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,217 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 237d1e590f534d84bfe7c745e4a9dca4 of snapshot=snaptb1-1676390334053 to region=70081217ed6e34352243aa2a9fed197b table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,217 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 02b28f40d4864ccd9260cf45d273018c of snapshot=snaptb1-1676390334053 to region=539ff23322a9c80991d27b3fbff0e4a6 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,218 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 0ff98c593797444dbfe2d530ee4ea760 of snapshot=snaptb1-1676390334053 to region=b3062a790bf6f13489f2b501c241d887 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,226 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 6ddde8eb26e04634a1bc07e0a58b1159 of snapshot=snaptb1-1676390334053 to region=a703922cf39fc2176a548eb4e7bceb40 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,227 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e6e75eef833c40099e6c444de9d5e253 of snapshot=snaptb1-1676390334053 to region=da0d94029320172fcd557222cbb7bf9d table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,229 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 9bb3e28f7c8440aa9b3c0b09a52a4e1b of snapshot=snaptb1-1676390334053 to region=0e3b60cf882c8a008dc41bf06dfa1855 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,230 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 736008cfbb8d47aea704b157afbae175 of snapshot=snaptb1-1676390334053 to region=b3062a790bf6f13489f2b501c241d887 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,231 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 9e9547d81f444025a05a320763442ca5 of snapshot=snaptb1-1676390334053 to region=70081217ed6e34352243aa2a9fed197b table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,233 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a84211fff4aa4e11b17352e147ba1969 of snapshot=snaptb1-1676390334053 to region=539ff23322a9c80991d27b3fbff0e4a6 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,233 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 8385c17038f646799efbd15afd668ef2 of snapshot=snaptb1-1676390334053 to region=a703922cf39fc2176a548eb4e7bceb40 table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,242 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:12,243 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390352243"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:12,243 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390352243"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:12,243 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390352243"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:12,243 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390352243"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:12,243 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390352243"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:12,243 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390352243"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:12,245 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}] 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390352244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390352244"}]},"ts":"1676390352244"} 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390352244"}]},"ts":"1676390352244"} 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390352244"}]},"ts":"1676390352244"} 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390352244"}]},"ts":"1676390352244"} 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390352244"}]},"ts":"1676390352244"} 2023-02-14 15:59:12,246 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390352244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390352244"}]},"ts":"1676390352244"} 2023-02-14 15:59:12,248 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-14 15:59:12,248 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-14 15:59:12,248 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}] 2023-02-14 15:59:12,248 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676390334053 table=testRestoreSnapshot-1676390334053 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676390334053 completed! 2023-02-14 15:59:12,250 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 ) in 101 msec 2023-02-14 15:59:12,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-14 15:59:12,403 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676390334053, procId: 147 completed 2023-02-14 15:59:12,403 INFO [Listener at localhost/39279] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:12,404 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676390334053-1676390351362" 2023-02-14 15:59:12,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676390334053-1676390351362 2023-02-14 15:59:12,407 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,407 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-14 15:59:12,411 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676390334053 2023-02-14 15:59:12,411 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390352411"}]},"ts":"1676390352411"} 2023-02-14 15:59:12,412 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:59:12,415 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:59:12,415 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:59:12,417 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:59:12,417 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:59:12,417 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:59:12,417 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:59:12,417 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:59:12,418 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:12,418 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:12,418 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:12,419 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:12,419 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:12,419 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:59:12,420 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-14 15:59:12,568 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:59:12,568 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:59:12,568 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:59:12,568 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:59:12,568 DEBUG [jenkins-hbase4:33899] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:59:12,572 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:12,572 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:12,572 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:12,572 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352572"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390352572"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390352572"}]},"ts":"1676390352572"} 2023-02-14 15:59:12,572 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:12,572 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:12,572 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390352571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390352571"}]},"ts":"1676390352571"} 2023-02-14 15:59:12,572 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390352571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390352571"}]},"ts":"1676390352571"} 2023-02-14 15:59:12,572 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390352571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390352571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390352571"}]},"ts":"1676390352571"} 2023-02-14 15:59:12,572 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390352571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390352571"}]},"ts":"1676390352571"} 2023-02-14 15:59:12,573 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:12,574 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=150, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:12,575 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=153, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:12,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=154, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:12,576 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=151, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:12,581 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:12,581 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390352581"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390352581"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390352581"}]},"ts":"1676390352581"} 2023-02-14 15:59:12,582 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=149, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:12,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-14 15:59:12,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:12,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:12,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:12,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:12,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:12,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:12,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,730 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,730 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,730 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:12,730 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:12,731 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:12,731 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:12,731 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:59:12,731 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:59:12,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:12,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:12,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:12,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,736 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,737 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:12,737 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:12,737 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:59:12,749 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:12,749 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:12,749 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:12,757 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:12,757 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:12,757 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:12,757 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:12,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,758 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:12,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,758 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:12,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:12,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:12,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62371436, jitterRate=-0.07059317827224731}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:12,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:12,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74613517, jitterRate=0.1118280440568924}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:12,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:12,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=158, masterSystemTime=1676390352732 2023-02-14 15:59:12,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=155, masterSystemTime=1676390352725 2023-02-14 15:59:12,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:12,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:12,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:12,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:12,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,769 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:12,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:12,769 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390352769"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390352769"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390352769"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390352769"}]},"ts":"1676390352769"} 2023-02-14 15:59:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:12,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:12,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:12,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:12,770 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:12,770 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352770"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390352770"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390352770"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390352770"}]},"ts":"1676390352770"} 2023-02-14 15:59:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:12,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:12,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60604947, jitterRate=-0.09691591560840607}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:12,773 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:12,773 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,774 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:12,775 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:12,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=157, masterSystemTime=1676390352726 2023-02-14 15:59:12,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=154 2023-02-14 15:59:12,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=154, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 196 msec 2023-02-14 15:59:12,775 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:12,775 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:12,775 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:59:12,801 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:59:12,803 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b384ceb to 127.0.0.1:55782 2023-02-14 15:59:12,803 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:12,804 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c2934c9 to 127.0.0.1:55782 2023-02-14 15:59:12,804 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 360 msec 2023-02-14 15:59:12,804 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:12,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-02-14 15:59:12,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 200 msec 2023-02-14 15:59:12,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:12,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:12,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:12,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:12,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:12,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,806 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:12,806 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352806"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390352806"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390352806"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390352806"}]},"ts":"1676390352806"} 2023-02-14 15:59:12,807 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 390 msec 2023-02-14 15:59:12,810 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=153 2023-02-14 15:59:12,810 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=153, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 233 msec 2023-02-14 15:59:12,811 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 395 msec 2023-02-14 15:59:12,814 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,816 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:12,816 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:12,818 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:59:12,824 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:12,836 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:12,844 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:12,845 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:12,845 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:12,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,849 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:12,850 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:12,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:12,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,853 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:12,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61084855, jitterRate=-0.08976472914218903}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:12,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:12,853 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:12,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=160, masterSystemTime=1676390352725 2023-02-14 15:59:12,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:12,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68398632, jitterRate=0.019219040870666504}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:12,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:12,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=159, masterSystemTime=1676390352732 2023-02-14 15:59:12,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:12,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71526447, jitterRate=0.06582711637020111}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:12,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:12,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=156, masterSystemTime=1676390352726 2023-02-14 15:59:12,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:12,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:12,862 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:12,862 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390352862"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390352862"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390352862"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390352862"}]},"ts":"1676390352862"} 2023-02-14 15:59:12,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:12,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:12,863 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:12,863 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352863"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390352863"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390352863"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390352863"}]},"ts":"1676390352863"} 2023-02-14 15:59:12,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:12,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:12,864 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:12,864 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390352864"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390352864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390352864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390352864"}]},"ts":"1676390352864"} 2023-02-14 15:59:12,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=149 2023-02-14 15:59:12,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=149, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 282 msec 2023-02-14 15:59:12,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=151 2023-02-14 15:59:12,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=151, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 289 msec 2023-02-14 15:59:12,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 453 msec 2023-02-14 15:59:12,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=150 2023-02-14 15:59:12,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=150, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 292 msec 2023-02-14 15:59:12,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 454 msec 2023-02-14 15:59:12,872 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-02-14 15:59:12,872 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 455 msec 2023-02-14 15:59:12,873 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390352873"}]},"ts":"1676390352873"} 2023-02-14 15:59:12,874 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:59:12,876 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676390334053' was successfully enabled. 2023-02-14 15:59:12,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676390334053 in 468 msec 2023-02-14 15:59:13,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-14 15:59:13,162 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 148 completed 2023-02-14 15:59:13,163 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(139): Connect 0x77b5b1f1 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:13,168 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1550b85e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:13,170 DEBUG [hconnection-0x2ed0e7fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:13,173 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45576, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:13,175 DEBUG [hconnection-0x2ed0e7fa-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:13,178 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,422 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:13,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,683 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:13,686 DEBUG [hconnection-0x2ed0e7fa-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:13,689 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51108, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:13,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,914 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:13,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:13,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,160 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:14,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,361 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:14,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:15,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,123 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 15:59:16,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,581 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676390334053' 2023-02-14 15:59:16,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:16,791 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:16,792 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:59:16,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:16,797 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390356796"}]},"ts":"1676390356796"} 2023-02-14 15:59:16,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-14 15:59:16,798 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:59:16,800 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:59:16,801 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:59:16,804 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:59:16,805 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:59:16,805 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:59:16,805 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:59:16,806 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:59:16,806 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:16,806 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:16,806 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:16,807 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390356806"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390356806"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390356806"}]},"ts":"1676390356806"} 2023-02-14 15:59:16,806 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:16,807 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390356806"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390356806"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390356806"}]},"ts":"1676390356806"} 2023-02-14 15:59:16,807 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:16,807 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390356806"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390356806"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390356806"}]},"ts":"1676390356806"} 2023-02-14 15:59:16,806 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390356806"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390356806"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390356806"}]},"ts":"1676390356806"} 2023-02-14 15:59:16,807 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390356807"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390356807"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390356807"}]},"ts":"1676390356807"} 2023-02-14 15:59:16,808 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=167, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:16,809 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=164, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:16,809 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=165, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:16,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=166, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:16,811 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=163, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:16,814 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:59:16,817 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:16,817 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390356817"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390356817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390356817"}]},"ts":"1676390356817"} 2023-02-14 15:59:16,818 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=162, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:16,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:16,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:59:16,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:16,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:16,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:59:16,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:16,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:16,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:59:16,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:16,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:16,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:59:16,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:16,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:16,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:59:16,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:16,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:16,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:59:16,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:16,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-14 15:59:16,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:16,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:16,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-14 15:59:16,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:16,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:16,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-14 15:59:16,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:16,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:16,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:16,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:16,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:59:16,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:16,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:16,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:59:16,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:16,983 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:59:16,986 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390356983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390356983"}]},"ts":"1676390356983"} 2023-02-14 15:59:16,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:16,989 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:16,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:59:16,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:16,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:16,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:59:16,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:16,990 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:59:16,990 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390356990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390356990"}]},"ts":"1676390356990"} 2023-02-14 15:59:16,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:16,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:16,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:59:16,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:16,995 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:59:16,995 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390356995"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390356995"}]},"ts":"1676390356995"} 2023-02-14 15:59:16,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:16,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:59:16,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:17,001 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=162 2023-02-14 15:59:17,001 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=162, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 177 msec 2023-02-14 15:59:17,001 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=164 2023-02-14 15:59:17,001 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=164, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 185 msec 2023-02-14 15:59:17,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=166 2023-02-14 15:59:17,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=166, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,40531,1676390331113 in 187 msec 2023-02-14 15:59:17,002 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 200 msec 2023-02-14 15:59:17,002 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 200 msec 2023-02-14 15:59:17,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 200 msec 2023-02-14 15:59:17,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-14 15:59:17,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-14 15:59:17,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:17,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:17,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:17,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:17,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-14 15:59:17,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:17,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:17,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:17,009 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:59:17,009 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390357009"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357009"}]},"ts":"1676390357009"} 2023-02-14 15:59:17,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:17,012 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:59:17,012 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390357012"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357012"}]},"ts":"1676390357012"} 2023-02-14 15:59:17,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:17,013 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:59:17,013 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390357013"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357013"}]},"ts":"1676390357013"} 2023-02-14 15:59:17,013 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=167 2023-02-14 15:59:17,013 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=167, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,43707,1676390331069 in 202 msec 2023-02-14 15:59:17,014 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 212 msec 2023-02-14 15:59:17,015 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=165 2023-02-14 15:59:17,015 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=165, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,37877,1676390331146 in 204 msec 2023-02-14 15:59:17,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=163 2023-02-14 15:59:17,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=163, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,40531,1676390331113 in 203 msec 2023-02-14 15:59:17,017 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 214 msec 2023-02-14 15:59:17,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=163, resume processing ppid=161 2023-02-14 15:59:17,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 215 msec 2023-02-14 15:59:17,018 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390357018"}]},"ts":"1676390357018"} 2023-02-14 15:59:17,020 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:59:17,021 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:59:17,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 230 msec 2023-02-14 15:59:17,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-14 15:59:17,049 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 161 completed 2023-02-14 15:59:17,053 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,061 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,063 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-14 15:59:17,071 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:17,071 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:17,071 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:17,071 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:17,071 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:17,071 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:17,074 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits] 2023-02-14 15:59:17,074 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits] 2023-02-14 15:59:17,074 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits] 2023-02-14 15:59:17,074 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits] 2023-02-14 15:59:17,075 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits] 2023-02-14 15:59:17,076 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits] 2023-02-14 15:59:17,078 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:17,079 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:17,080 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:17,080 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:17,080 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:17,081 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:17,082 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:17,082 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:17,083 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:17,083 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:17,083 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:17,086 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:17,090 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/27.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/27.seqid 2023-02-14 15:59:17,091 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/27.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/27.seqid 2023-02-14 15:59:17,091 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:17,091 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/27.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/27.seqid 2023-02-14 15:59:17,091 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/27.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/27.seqid 2023-02-14 15:59:17,091 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:17,092 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/27.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/27.seqid 2023-02-14 15:59:17,092 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:17,092 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:17,092 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:17,093 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/27.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/27.seqid 2023-02-14 15:59:17,093 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:17,094 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676390334053 regions 2023-02-14 15:59:17,094 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f 2023-02-14 15:59:17,095 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf] 2023-02-14 15:59:17,097 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:59:17,098 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e 2023-02-14 15:59:17,099 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:59:17,100 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce 2023-02-14 15:59:17,102 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:59:17,102 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 2023-02-14 15:59:17,103 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:59:17,105 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f 2023-02-14 15:59:17,106 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:59:17,107 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e 2023-02-14 15:59:17,108 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c 2023-02-14 15:59:17,110 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:59:17,110 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f 2023-02-14 15:59:17,112 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,135 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676390334053 from hbase:meta 2023-02-14 15:59:17,137 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676390334053' descriptor. 2023-02-14 15:59:17,138 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,138 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676390334053' from region states. 2023-02-14 15:59:17,138 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390357138"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390357138"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390357138"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390357138"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390357138"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390357138"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,140 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-14 15:59:17,141 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}] 2023-02-14 15:59:17,141 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676390334053' as deleted. 2023-02-14 15:59:17,141 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676390357141"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:17,142 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676390334053 state from META 2023-02-14 15:59:17,145 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,146 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 in 91 msec 2023-02-14 15:59:17,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-14 15:59:17,316 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 174 completed 2023-02-14 15:59:17,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 ) 2023-02-14 15:59:17,350 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676390334053 as table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-14 15:59:17,365 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/.tabledesc/.tableinfo.0000000001 2023-02-14 15:59:17,367 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:17,367 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,368 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,371 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,374 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,376 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,378 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,380 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,382 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,384 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,386 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,388 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,390 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,392 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,395 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:17,395 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:17,395 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=0e3b60cf882c8a008dc41bf06dfa1855 as 0e3b60cf882c8a008dc41bf06dfa1855 in snapshot snaptb1-1676390334053 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=b3062a790bf6f13489f2b501c241d887 as b3062a790bf6f13489f2b501c241d887 in snapshot snaptb1-1676390334053 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=da0d94029320172fcd557222cbb7bf9d as da0d94029320172fcd557222cbb7bf9d in snapshot snaptb1-1676390334053 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=a703922cf39fc2176a548eb4e7bceb40 as a703922cf39fc2176a548eb4e7bceb40 in snapshot snaptb1-1676390334053 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=539ff23322a9c80991d27b3fbff0e4a6 as 539ff23322a9c80991d27b3fbff0e4a6 in snapshot snaptb1-1676390334053 2023-02-14 15:59:17,396 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=70081217ed6e34352243aa2a9fed197b as 70081217ed6e34352243aa2a9fed197b in snapshot snaptb1-1676390334053 2023-02-14 15:59:17,398 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:17,398 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:17,398 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:17,398 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:17,398 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:17,398 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676390334053', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:17,432 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:17,432 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 6d30ae786a014cb4b9f248d9f924f34c from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,433 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:17,433 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 0ff98c593797444dbfe2d530ee4ea760 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,433 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:17,433 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 6ddde8eb26e04634a1bc07e0a58b1159 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,444 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 736008cfbb8d47aea704b157afbae175 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,444 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 9bb3e28f7c8440aa9b3c0b09a52a4e1b from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,445 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 8385c17038f646799efbd15afd668ef2 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,446 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:17,446 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 02b28f40d4864ccd9260cf45d273018c from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,446 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:17,446 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d0cbb40341b14f699c493a7ab9a2b0b1 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,448 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:59:17,448 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:59:17,448 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:17,448 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:17,448 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:17,448 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:59:17,449 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:17,449 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:17,449 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:17,449 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:17,449 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:17,450 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e6e75eef833c40099e6c444de9d5e253 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,450 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a84211fff4aa4e11b17352e147ba1969 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,452 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:59:17,452 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:17,452 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:17,452 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:59:17,452 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:59:17,452 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:17,452 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:17,453 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:17,453 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:17,453 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:59:17,453 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:17,453 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:17,453 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:17,453 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-14 15:59:17,834 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:17,835 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 237d1e590f534d84bfe7c745e4a9dca4 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,837 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 9e9547d81f444025a05a320763442ca5 from cloned region in snapshot snaptb1-1676390334053 to table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:17,840 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:59:17,840 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:17,840 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:17,840 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:59:17,840 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:17,840 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:17,840 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:17,840 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 2023-02-14 15:59:17,840 INFO [PEWorker-4] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676390334053 on table=testRestoreSnapshot-1676390334053 completed! 2023-02-14 15:59:17,846 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390357845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357845"}]},"ts":"1676390357845"} 2023-02-14 15:59:17,846 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390357845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357845"}]},"ts":"1676390357845"} 2023-02-14 15:59:17,846 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390357845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357845"}]},"ts":"1676390357845"} 2023-02-14 15:59:17,846 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390357845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357845"}]},"ts":"1676390357845"} 2023-02-14 15:59:17,846 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390357845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357845"}]},"ts":"1676390357845"} 2023-02-14 15:59:17,846 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390357845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390357845"}]},"ts":"1676390357845"} 2023-02-14 15:59:17,848 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-14 15:59:17,850 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390357850"}]},"ts":"1676390357850"} 2023-02-14 15:59:17,851 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLING in hbase:meta 2023-02-14 15:59:17,856 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:59:17,856 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:59:17,856 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:59:17,856 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:59:17,856 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:59:17,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN}] 2023-02-14 15:59:17,858 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN 2023-02-14 15:59:17,858 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN 2023-02-14 15:59:17,858 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN 2023-02-14 15:59:17,858 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN 2023-02-14 15:59:17,859 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:59:17,859 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN 2023-02-14 15:59:17,859 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=false 2023-02-14 15:59:17,859 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=false 2023-02-14 15:59:17,859 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=false 2023-02-14 15:59:17,860 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:59:17,860 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN 2023-02-14 15:59:17,861 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=false 2023-02-14 15:59:18,009 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:59:18,014 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:18,014 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:18,014 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390358014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390358014"}]},"ts":"1676390358014"} 2023-02-14 15:59:18,014 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390358014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390358014"}]},"ts":"1676390358014"} 2023-02-14 15:59:18,014 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:18,014 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:18,015 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390358014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390358014"}]},"ts":"1676390358014"} 2023-02-14 15:59:18,015 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390358014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390358014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390358014"}]},"ts":"1676390358014"} 2023-02-14 15:59:18,014 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:18,016 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390358014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390358014"}]},"ts":"1676390358014"} 2023-02-14 15:59:18,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=180, state=RUNNABLE; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:18,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=178, state=RUNNABLE; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:18,021 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=179, state=RUNNABLE; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:18,023 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=181, state=RUNNABLE; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:18,024 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=177, state=RUNNABLE; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:18,024 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:18,024 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390358024"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390358024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390358024"}]},"ts":"1676390358024"} 2023-02-14 15:59:18,026 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=176, state=RUNNABLE; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:18,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-14 15:59:18,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:18,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:18,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:18,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,176 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:18,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:18,176 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:18,176 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf 2023-02-14 15:59:18,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:18,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,177 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b3062a790bf6f13489f2b501c241d887 columnFamilyName cf 2023-02-14 15:59:18,178 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,178 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:18,179 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf 2023-02-14 15:59:18,179 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a703922cf39fc2176a548eb4e7bceb40 columnFamilyName cf 2023-02-14 15:59:18,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,193 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,194 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:18,194 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:18,194 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:18,195 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf 2023-02-14 15:59:18,195 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70081217ed6e34352243aa2a9fed197b columnFamilyName cf 2023-02-14 15:59:18,205 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:18,205 DEBUG [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:18,205 INFO [StoreOpener-b3062a790bf6f13489f2b501c241d887-1] regionserver.HStore(310): Store=b3062a790bf6f13489f2b501c241d887/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:18,205 DEBUG [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:18,205 INFO [StoreOpener-a703922cf39fc2176a548eb4e7bceb40-1] regionserver.HStore(310): Store=a703922cf39fc2176a548eb4e7bceb40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:18,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:18,213 DEBUG [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-14 15:59:18,213 INFO [StoreOpener-70081217ed6e34352243aa2a9fed197b-1] regionserver.HStore(310): Store=70081217ed6e34352243aa2a9fed197b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:18,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-14 15:59:18,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b3062a790bf6f13489f2b501c241d887; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63109455, jitterRate=-0.05959583818912506}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:18,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:18,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887., pid=183, masterSystemTime=1676390358171 2023-02-14 15:59:18,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a703922cf39fc2176a548eb4e7bceb40; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65013211, jitterRate=-0.031227663159370422}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:18,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:18,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40., pid=184, masterSystemTime=1676390358174 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:18,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:18,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:18,217 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:18,217 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358217"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390358217"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390358217"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390358217"}]},"ts":"1676390358217"} 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:18,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:18,218 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:18,218 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358218"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390358218"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390358218"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390358218"}]},"ts":"1676390358218"} 2023-02-14 15:59:18,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:18,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:18,219 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,220 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,220 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:18,221 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf 2023-02-14 15:59:18,221 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:18,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-14 15:59:18,221 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf 2023-02-14 15:59:18,222 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539ff23322a9c80991d27b3fbff0e4a6 columnFamilyName cf 2023-02-14 15:59:18,222 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e3b60cf882c8a008dc41bf06dfa1855 columnFamilyName cf 2023-02-14 15:59:18,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=178 2023-02-14 15:59:18,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=178, state=SUCCESS; OpenRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 202 msec 2023-02-14 15:59:18,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=179 2023-02-14 15:59:18,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=179, state=SUCCESS; OpenRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,40531,1676390331113 in 200 msec 2023-02-14 15:59:18,224 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, ASSIGN in 367 msec 2023-02-14 15:59:18,224 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, ASSIGN in 367 msec 2023-02-14 15:59:18,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 70081217ed6e34352243aa2a9fed197b; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74008621, jitterRate=0.10281439125537872}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:18,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:18,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b., pid=186, masterSystemTime=1676390358176 2023-02-14 15:59:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:18,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:18,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:18,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676390334053 da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:18,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,229 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:18,229 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358229"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390358229"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390358229"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390358229"}]},"ts":"1676390358229"} 2023-02-14 15:59:18,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=177 2023-02-14 15:59:18,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=177, state=SUCCESS; OpenRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146 in 207 msec 2023-02-14 15:59:18,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, ASSIGN in 376 msec 2023-02-14 15:59:18,238 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:18,244 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,245 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:18,245 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf 2023-02-14 15:59:18,246 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da0d94029320172fcd557222cbb7bf9d columnFamilyName cf 2023-02-14 15:59:18,247 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:18,250 DEBUG [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:18,251 INFO [StoreOpener-539ff23322a9c80991d27b3fbff0e4a6-1] regionserver.HStore(310): Store=539ff23322a9c80991d27b3fbff0e4a6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:18,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,252 DEBUG [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:18,253 INFO [StoreOpener-0e3b60cf882c8a008dc41bf06dfa1855-1] regionserver.HStore(310): Store=0e3b60cf882c8a008dc41bf06dfa1855/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:18,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:18,255 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:18,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:18,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-14 15:59:18,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 539ff23322a9c80991d27b3fbff0e4a6; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58755128, jitterRate=-0.12448036670684814}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:18,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:18,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6., pid=182, masterSystemTime=1676390358171 2023-02-14 15:59:18,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:18,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:18,264 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:18,265 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390358264"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390358264"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390358264"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390358264"}]},"ts":"1676390358264"} 2023-02-14 15:59:18,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=180 2023-02-14 15:59:18,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=180, state=SUCCESS; OpenRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,43707,1676390331069 in 249 msec 2023-02-14 15:59:18,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-14 15:59:18,268 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, ASSIGN in 411 msec 2023-02-14 15:59:18,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0e3b60cf882c8a008dc41bf06dfa1855; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69945137, jitterRate=0.042263761162757874}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:18,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:18,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855., pid=185, masterSystemTime=1676390358174 2023-02-14 15:59:18,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:18,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:18,270 DEBUG [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:18,270 INFO [StoreOpener-da0d94029320172fcd557222cbb7bf9d-1] regionserver.HStore(310): Store=da0d94029320172fcd557222cbb7bf9d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:18,271 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:18,271 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390358270"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390358270"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390358270"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390358270"}]},"ts":"1676390358270"} 2023-02-14 15:59:18,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,274 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=181 2023-02-14 15:59:18,274 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=181, state=SUCCESS; OpenRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113 in 249 msec 2023-02-14 15:59:18,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:18,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, ASSIGN in 418 msec 2023-02-14 15:59:18,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-14 15:59:18,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened da0d94029320172fcd557222cbb7bf9d; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65816394, jitterRate=-0.01925930380821228}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:18,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:18,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d., pid=187, masterSystemTime=1676390358176 2023-02-14 15:59:18,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:18,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:18,279 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:18,279 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390358279"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390358279"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390358279"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390358279"}]},"ts":"1676390358279"} 2023-02-14 15:59:18,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=176 2023-02-14 15:59:18,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=176, state=SUCCESS; OpenRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 255 msec 2023-02-14 15:59:18,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=176, resume processing ppid=175 2023-02-14 15:59:18,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, ASSIGN in 426 msec 2023-02-14 15:59:18,289 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390358289"}]},"ts":"1676390358289"} 2023-02-14 15:59:18,292 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=ENABLED in hbase:meta 2023-02-14 15:59:18,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676390334053 snapshot=name: "snaptb1-1676390334053" table: "testRestoreSnapshot-1676390334053" creation_time: 1676390346261 type: DISABLED version: 2 ttl: 0 ) in 958 msec 2023-02-14 15:59:18,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-14 15:59:18,856 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676390334053, procId: 175 completed 2023-02-14 15:59:18,857 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(139): Connect 0x4672389c to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:18,865 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ff51970, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:18,867 DEBUG [hconnection-0x4c9ce563-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:18,869 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51034, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:18,872 DEBUG [hconnection-0x4c9ce563-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:18,874 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50888, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:18,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,122 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:19,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,356 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:19,363 DEBUG [hconnection-0x4c9ce563-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:19,366 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:19,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,551 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,727 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:19,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,897 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:19,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:19,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,178 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77b5b1f1 to 127.0.0.1:55782 2023-02-14 15:59:20,178 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:21,863 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676390334053 2023-02-14 15:59:21,864 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676390334053 2023-02-14 15:59:21,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:21,868 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390361868"}]},"ts":"1676390361868"} 2023-02-14 15:59:21,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-14 15:59:21,871 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLING in hbase:meta 2023-02-14 15:59:21,872 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676390334053 to state=DISABLING 2023-02-14 15:59:21,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN}] 2023-02-14 15:59:21,875 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN 2023-02-14 15:59:21,875 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN 2023-02-14 15:59:21,875 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN 2023-02-14 15:59:21,875 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN 2023-02-14 15:59:21,875 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN 2023-02-14 15:59:21,876 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:21,876 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:21,876 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:21,876 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390361876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390361876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390361876"}]},"ts":"1676390361876"} 2023-02-14 15:59:21,876 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390361876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390361876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390361876"}]},"ts":"1676390361876"} 2023-02-14 15:59:21,876 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:21,876 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390361876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390361876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390361876"}]},"ts":"1676390361876"} 2023-02-14 15:59:21,877 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390361876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390361876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390361876"}]},"ts":"1676390361876"} 2023-02-14 15:59:21,876 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:21,877 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390361876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390361876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390361876"}]},"ts":"1676390361876"} 2023-02-14 15:59:21,878 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=189, state=RUNNABLE; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:21,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=190, state=RUNNABLE; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:21,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=191, state=RUNNABLE; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:21,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=194, state=RUNNABLE; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:21,888 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN 2023-02-14 15:59:21,888 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=192, state=RUNNABLE; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:21,889 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:21,889 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390361889"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390361889"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390361889"}]},"ts":"1676390361889"} 2023-02-14 15:59:21,891 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:22,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:22,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 70081217ed6e34352243aa2a9fed197b, disabling compactions & flushes 2023-02-14 15:59:22,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:22,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:22,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. after waiting 0 ms 2023-02-14 15:59:22,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:22,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:22,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 539ff23322a9c80991d27b3fbff0e4a6, disabling compactions & flushes 2023-02-14 15:59:22,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:22,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:22,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. after waiting 0 ms 2023-02-14 15:59:22,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:22,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:22,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a703922cf39fc2176a548eb4e7bceb40, disabling compactions & flushes 2023-02-14 15:59:22,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:22,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:22,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. after waiting 0 ms 2023-02-14 15:59:22,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:22,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-14 15:59:22,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b. 2023-02-14 15:59:22,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 70081217ed6e34352243aa2a9fed197b: 2023-02-14 15:59:22,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:22,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:22,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing da0d94029320172fcd557222cbb7bf9d, disabling compactions & flushes 2023-02-14 15:59:22,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:22,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:22,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. after waiting 0 ms 2023-02-14 15:59:22,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:22,051 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=70081217ed6e34352243aa2a9fed197b, regionState=CLOSED 2023-02-14 15:59:22,051 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390362051"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362051"}]},"ts":"1676390362051"} 2023-02-14 15:59:22,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-14 15:59:22,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40. 2023-02-14 15:59:22,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-14 15:59:22,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a703922cf39fc2176a548eb4e7bceb40: 2023-02-14 15:59:22,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6. 2023-02-14 15:59:22,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 539ff23322a9c80991d27b3fbff0e4a6: 2023-02-14 15:59:22,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:22,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:22,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0e3b60cf882c8a008dc41bf06dfa1855, disabling compactions & flushes 2023-02-14 15:59:22,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:22,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:22,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. after waiting 0 ms 2023-02-14 15:59:22,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:22,063 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=190 2023-02-14 15:59:22,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=190, state=SUCCESS; CloseRegionProcedure 70081217ed6e34352243aa2a9fed197b, server=jenkins-hbase4.apache.org,37877,1676390331146 in 182 msec 2023-02-14 15:59:22,064 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=a703922cf39fc2176a548eb4e7bceb40, regionState=CLOSED 2023-02-14 15:59:22,064 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390362064"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362064"}]},"ts":"1676390362064"} 2023-02-14 15:59:22,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:22,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:22,065 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=539ff23322a9c80991d27b3fbff0e4a6, regionState=CLOSED 2023-02-14 15:59:22,065 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390362065"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362065"}]},"ts":"1676390362065"} 2023-02-14 15:59:22,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b3062a790bf6f13489f2b501c241d887, disabling compactions & flushes 2023-02-14 15:59:22,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-14 15:59:22,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:22,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=70081217ed6e34352243aa2a9fed197b, UNASSIGN in 191 msec 2023-02-14 15:59:22,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:22,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. after waiting 0 ms 2023-02-14 15:59:22,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:22,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d. 2023-02-14 15:59:22,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for da0d94029320172fcd557222cbb7bf9d: 2023-02-14 15:59:22,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:22,070 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=da0d94029320172fcd557222cbb7bf9d, regionState=CLOSED 2023-02-14 15:59:22,070 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390362070"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362070"}]},"ts":"1676390362070"} 2023-02-14 15:59:22,071 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=192 2023-02-14 15:59:22,071 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=192, state=SUCCESS; CloseRegionProcedure a703922cf39fc2176a548eb4e7bceb40, server=jenkins-hbase4.apache.org,40531,1676390331113 in 179 msec 2023-02-14 15:59:22,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-14 15:59:22,072 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-14 15:59:22,072 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure 539ff23322a9c80991d27b3fbff0e4a6, server=jenkins-hbase4.apache.org,43707,1676390331069 in 178 msec 2023-02-14 15:59:22,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855. 2023-02-14 15:59:22,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0e3b60cf882c8a008dc41bf06dfa1855: 2023-02-14 15:59:22,073 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=a703922cf39fc2176a548eb4e7bceb40, UNASSIGN in 198 msec 2023-02-14 15:59:22,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-14 15:59:22,074 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887. 2023-02-14 15:59:22,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b3062a790bf6f13489f2b501c241d887: 2023-02-14 15:59:22,074 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=539ff23322a9c80991d27b3fbff0e4a6, UNASSIGN in 199 msec 2023-02-14 15:59:22,074 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:22,075 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=189 2023-02-14 15:59:22,075 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=0e3b60cf882c8a008dc41bf06dfa1855, regionState=CLOSED 2023-02-14 15:59:22,075 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=189, state=SUCCESS; CloseRegionProcedure da0d94029320172fcd557222cbb7bf9d, server=jenkins-hbase4.apache.org,37877,1676390331146 in 194 msec 2023-02-14 15:59:22,075 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676390362075"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362075"}]},"ts":"1676390362075"} 2023-02-14 15:59:22,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:22,076 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=b3062a790bf6f13489f2b501c241d887, regionState=CLOSED 2023-02-14 15:59:22,076 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676390362076"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362076"}]},"ts":"1676390362076"} 2023-02-14 15:59:22,077 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=da0d94029320172fcd557222cbb7bf9d, UNASSIGN in 202 msec 2023-02-14 15:59:22,078 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=194 2023-02-14 15:59:22,078 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=194, state=SUCCESS; CloseRegionProcedure 0e3b60cf882c8a008dc41bf06dfa1855, server=jenkins-hbase4.apache.org,40531,1676390331113 in 193 msec 2023-02-14 15:59:22,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=191 2023-02-14 15:59:22,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=191, state=SUCCESS; CloseRegionProcedure b3062a790bf6f13489f2b501c241d887, server=jenkins-hbase4.apache.org,43707,1676390331069 in 198 msec 2023-02-14 15:59:22,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=0e3b60cf882c8a008dc41bf06dfa1855, UNASSIGN in 205 msec 2023-02-14 15:59:22,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=191, resume processing ppid=188 2023-02-14 15:59:22,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676390334053, region=b3062a790bf6f13489f2b501c241d887, UNASSIGN in 206 msec 2023-02-14 15:59:22,082 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390362082"}]},"ts":"1676390362082"} 2023-02-14 15:59:22,083 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676390334053, state=DISABLED in hbase:meta 2023-02-14 15:59:22,085 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676390334053 to state=DISABLED 2023-02-14 15:59:22,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676390334053 in 222 msec 2023-02-14 15:59:22,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-14 15:59:22,120 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 188 completed 2023-02-14 15:59:22,121 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,123 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,124 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-14 15:59:22,132 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:22,132 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:22,132 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:22,132 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:22,132 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:22,132 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:22,134 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits] 2023-02-14 15:59:22,135 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits] 2023-02-14 15:59:22,135 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits] 2023-02-14 15:59:22,135 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits] 2023-02-14 15:59:22,135 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits] 2023-02-14 15:59:22,136 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits] 2023-02-14 15:59:22,140 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357079 currentMtime: 1676390357440 2023-02-14 15:59:22,142 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760.1676390362134. 2023-02-14 15:59:22,142 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357080 currentMtime: 1676390357450 2023-02-14 15:59:22,142 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357078 currentMtime: 1676390357444 2023-02-14 15:59:22,143 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357077 currentMtime: 1676390357837 2023-02-14 15:59:22,143 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159.1676390362135. 2023-02-14 15:59:22,144 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1.1676390362135. 2023-02-14 15:59:22,144 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-0ff98c593797444dbfe2d530ee4ea760 2023-02-14 15:59:22,144 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4.1676390362135. 2023-02-14 15:59:22,145 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357078 currentMtime: 1676390357450 2023-02-14 15:59:22,145 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357079 currentMtime: 1676390357440 2023-02-14 15:59:22,145 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-d0cbb40341b14f699c493a7ab9a2b0b1 2023-02-14 15:59:22,146 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c.1676390362135. 2023-02-14 15:59:22,146 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-6ddde8eb26e04634a1bc07e0a58b1159 2023-02-14 15:59:22,146 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c.1676390362136. 2023-02-14 15:59:22,146 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-237d1e590f534d84bfe7c745e4a9dca4 2023-02-14 15:59:22,147 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357081 currentMtime: 1676390357448 2023-02-14 15:59:22,147 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-02b28f40d4864ccd9260cf45d273018c 2023-02-14 15:59:22,148 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357085 currentMtime: 1676390357452 2023-02-14 15:59:22,148 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175.1676390362134. 2023-02-14 15:59:22,148 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253.1676390362135. 2023-02-14 15:59:22,148 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-6d30ae786a014cb4b9f248d9f924f34c 2023-02-14 15:59:22,149 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357081 currentMtime: 1676390357448 2023-02-14 15:59:22,150 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/cf/testRestoreSnapshot-1676390334053=b3062a790bf6f13489f2b501c241d887-736008cfbb8d47aea704b157afbae175 2023-02-14 15:59:22,150 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/cf/testRestoreSnapshot-1676390334053=da0d94029320172fcd557222cbb7bf9d-e6e75eef833c40099e6c444de9d5e253 2023-02-14 15:59:22,151 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357080 currentMtime: 1676390357839 2023-02-14 15:59:22,151 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2.1676390362135. 2023-02-14 15:59:22,151 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357080 currentMtime: 1676390357452 2023-02-14 15:59:22,152 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969.1676390362135. 2023-02-14 15:59:22,152 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5.1676390362135. 2023-02-14 15:59:22,152 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357082 currentMtime: 1676390357448 2023-02-14 15:59:22,152 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/cf/testRestoreSnapshot-1676390334053=a703922cf39fc2176a548eb4e7bceb40-8385c17038f646799efbd15afd668ef2 2023-02-14 15:59:22,153 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b.1676390362136. 2023-02-14 15:59:22,153 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/cf/testRestoreSnapshot-1676390334053=539ff23322a9c80991d27b3fbff0e4a6-a84211fff4aa4e11b17352e147ba1969 2023-02-14 15:59:22,154 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/cf/testRestoreSnapshot-1676390334053=70081217ed6e34352243aa2a9fed197b-9e9547d81f444025a05a320763442ca5 2023-02-14 15:59:22,155 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/cf/testRestoreSnapshot-1676390334053=0e3b60cf882c8a008dc41bf06dfa1855-9bb3e28f7c8440aa9b3c0b09a52a4e1b 2023-02-14 15:59:22,156 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887/recovered.edits/18.seqid 2023-02-14 15:59:22,157 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d/recovered.edits/18.seqid 2023-02-14 15:59:22,157 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/b3062a790bf6f13489f2b501c241d887 2023-02-14 15:59:22,158 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/da0d94029320172fcd557222cbb7bf9d 2023-02-14 15:59:22,159 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40/recovered.edits/18.seqid 2023-02-14 15:59:22,160 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/a703922cf39fc2176a548eb4e7bceb40 2023-02-14 15:59:22,160 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6/recovered.edits/18.seqid 2023-02-14 15:59:22,160 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855/recovered.edits/18.seqid 2023-02-14 15:59:22,160 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b/recovered.edits/18.seqid 2023-02-14 15:59:22,160 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/539ff23322a9c80991d27b3fbff0e4a6 2023-02-14 15:59:22,161 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/0e3b60cf882c8a008dc41bf06dfa1855 2023-02-14 15:59:22,161 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testRestoreSnapshot-1676390334053/70081217ed6e34352243aa2a9fed197b 2023-02-14 15:59:22,161 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676390334053 regions 2023-02-14 15:59:22,161 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f 2023-02-14 15:59:22,162 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf] 2023-02-14 15:59:22,164 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357097 currentMtime: 1676390357371 2023-02-14 15:59:22,164 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5.1676390362162. 2023-02-14 15:59:22,165 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c202302140d5b02bd8e3f43d4ad418cce3e0590c5 2023-02-14 15:59:22,166 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357098 currentMtime: 1676390357373 2023-02-14 15:59:22,166 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e.1676390362162. 2023-02-14 15:59:22,167 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-a87ff679a2f3e71d9181a67b7542122c20230214ed341dac552447bb8a33e30cf9a87f5e 2023-02-14 15:59:22,168 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357099 currentMtime: 1676390357376 2023-02-14 15:59:22,169 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337.1676390362162. 2023-02-14 15:59:22,169 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b202302149b6e127074a04b4a977ca1f7b0451337 2023-02-14 15:59:22,170 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357100 currentMtime: 1676390357378 2023-02-14 15:59:22,171 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce.1676390362162. 2023-02-14 15:59:22,171 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c4ca4238a0b923820dcc509a6f75849b20230214c58d4fe3ec39416c8a79ccb52b0629ce 2023-02-14 15:59:22,172 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357101 currentMtime: 1676390357380 2023-02-14 15:59:22,173 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802.1676390362162. 2023-02-14 15:59:22,173 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c2023021485c2f0424f0349638277d5def39e6802 2023-02-14 15:59:22,174 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357102 currentMtime: 1676390357382 2023-02-14 15:59:22,174 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0.1676390362162. 2023-02-14 15:59:22,175 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-c81e728d9d4c2f636f067f89cc14862c20230214df285e3ab1b24c4bbe8a571db7994ab0 2023-02-14 15:59:22,176 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357103 currentMtime: 1676390357384 2023-02-14 15:59:22,176 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870.1676390362162. 2023-02-14 15:59:22,177 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e202302141912b290d99745f994c4acd5d3b43870 2023-02-14 15:59:22,178 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357104 currentMtime: 1676390357386 2023-02-14 15:59:22,178 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f.1676390362162. 2023-02-14 15:59:22,179 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-d41d8cd98f00b204e9800998ecf8427e20230214272bf5a5517144ed96ce1c3c2db4921f 2023-02-14 15:59:22,180 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357105 currentMtime: 1676390357388 2023-02-14 15:59:22,180 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45.1676390362162. 2023-02-14 15:59:22,181 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021447db5b7751cf4788a3da011845c50d45 2023-02-14 15:59:22,182 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357106 currentMtime: 1676390357389 2023-02-14 15:59:22,183 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e.1676390362162. 2023-02-14 15:59:22,183 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-e4da3b7fbbce2345d7772b0674a318d52023021490fc326e3cd04a5bbd1b31e7823d4f8e 2023-02-14 15:59:22,185 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357108 currentMtime: 1676390357391 2023-02-14 15:59:22,185 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c.1676390362162. 2023-02-14 15:59:22,186 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021428de64a963eb467d93672a676b20aa0c 2023-02-14 15:59:22,187 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc. archiveLen: 0 currentLen: 0 archiveMtime: 1676390357109 currentMtime: 1676390357395 2023-02-14 15:59:22,187 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc.1676390362162. 2023-02-14 15:59:22,188 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f/cf/testRestoreSnapshot-1676390334053=8852f07d6ff59754d6b918836384d34f-eccbc87e4b5ce2fe28308fd9f2a7baf320230214c44f35f7546a49c9b0380751524499cc 2023-02-14 15:59:22,188 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testRestoreSnapshot-1676390334053/8852f07d6ff59754d6b918836384d34f 2023-02-14 15:59:22,190 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,193 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676390334053 from hbase:meta 2023-02-14 15:59:22,195 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676390334053' descriptor. 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676390334053' from region states. 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390362196"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390362196"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390362196"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390362196"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390362196"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,196 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390362196"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,198 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-14 15:59:22,198 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => da0d94029320172fcd557222cbb7bf9d, NAME => 'testRestoreSnapshot-1676390334053,,1676390334081.da0d94029320172fcd557222cbb7bf9d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 70081217ed6e34352243aa2a9fed197b, NAME => 'testRestoreSnapshot-1676390334053,1,1676390334081.70081217ed6e34352243aa2a9fed197b.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => b3062a790bf6f13489f2b501c241d887, NAME => 'testRestoreSnapshot-1676390334053,2,1676390334081.b3062a790bf6f13489f2b501c241d887.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => a703922cf39fc2176a548eb4e7bceb40, NAME => 'testRestoreSnapshot-1676390334053,3,1676390334081.a703922cf39fc2176a548eb4e7bceb40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 539ff23322a9c80991d27b3fbff0e4a6, NAME => 'testRestoreSnapshot-1676390334053,4,1676390334081.539ff23322a9c80991d27b3fbff0e4a6.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 0e3b60cf882c8a008dc41bf06dfa1855, NAME => 'testRestoreSnapshot-1676390334053,5,1676390334081.0e3b60cf882c8a008dc41bf06dfa1855.', STARTKEY => '5', ENDKEY => ''}] 2023-02-14 15:59:22,198 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676390334053' as deleted. 2023-02-14 15:59:22,198 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676390334053","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676390362198"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:22,199 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676390334053 state from META 2023-02-14 15:59:22,201 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 2023-02-14 15:59:22,202 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676390334053 in 80 msec 2023-02-14 15:59:22,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-14 15:59:22,376 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676390334053, procId: 201 completed 2023-02-14 15:59:22,385 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676390334053" 2023-02-14 15:59:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676390334053 2023-02-14 15:59:22,390 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676390334053" 2023-02-14 15:59:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676390334053 2023-02-14 15:59:22,394 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676390334053" 2023-02-14 15:59:22,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676390334053 2023-02-14 15:59:22,441 INFO [Listener at localhost/39279] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=412 (was 358) Potentially hanging thread: hconnection-0x5c7402a7-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1555434563_15 at /127.0.0.1:33514 [Waiting for operation #530] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55782@0x4672389c-SendThread(127.0.0.1:55782) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1555434563_15 at /127.0.0.1:51352 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1555434563_15 at /127.0.0.1:33530 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5c7402a7-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55782@0x4672389c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/1221834836.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1675794290_15 at /127.0.0.1:41694 [Waiting for operation #493] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1555434563_15 at /127.0.0.1:33500 [Waiting for operation #528] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5c7402a7-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55782@0x4672389c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1675794290_15 at /127.0.0.1:41690 [Waiting for operation #518] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1555434563_15 at /127.0.0.1:33532 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5c7402a7-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=783 (was 688) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=328 (was 262) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4450 (was 4294) - AvailableMemoryMB LEAK? - 2023-02-14 15:59:22,465 INFO [Listener at localhost/39279] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=412, OpenFileDescriptor=783, MaxFileDescriptor=60000, SystemLoadAverage=328, ProcessCount=168, AvailableMemoryMB=4449 2023-02-14 15:59:22,468 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-14 15:59:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:22,472 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 15:59:22,472 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676390362465" procId is: 202 2023-02-14 15:59:22,473 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 15:59:22,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-14 15:59:22,481 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,481 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,481 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,481 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,481 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,481 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,482 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f empty. 2023-02-14 15:59:22,482 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b empty. 2023-02-14 15:59:22,482 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 empty. 2023-02-14 15:59:22,482 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 empty. 2023-02-14 15:59:22,482 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 15:59:22,483 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a empty. 2023-02-14 15:59:22,483 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,483 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb empty. 2023-02-14 15:59:22,484 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,484 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,483 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,484 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,484 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,484 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676390362465 regions 2023-02-14 15:59:22,521 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/.tabledesc/.tableinfo.0000000001 2023-02-14 15:59:22,525 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(7671): creating {ENCODED => 69a0168d9f604b96943e74817bde200a, NAME => 'testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:22,525 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(7671): creating {ENCODED => 0d736737d4ae8d06714c55079f6c2b15, NAME => 'testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:22,525 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(7671): creating {ENCODED => 10c0bb63b1c83f26d2954d286d39b71f, NAME => 'testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:22,556 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,556 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1603): Closing 0d736737d4ae8d06714c55079f6c2b15, disabling compactions & flushes 2023-02-14 15:59:22,556 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,556 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,556 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. after waiting 0 ms 2023-02-14 15:59:22,556 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,556 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,557 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1557): Region close journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:22,557 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(7671): creating {ENCODED => 7ae37dd64d13c37bf459120525893ee5, NAME => 'testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:22,584 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,584 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1603): Closing 10c0bb63b1c83f26d2954d286d39b71f, disabling compactions & flushes 2023-02-14 15:59:22,584 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:22,584 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:22,584 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. after waiting 0 ms 2023-02-14 15:59:22,584 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:22,584 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:22,585 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1557): Region close journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:22,585 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(7671): creating {ENCODED => 7b10d6f25ee192486c2eccbdfd488bbb, NAME => 'testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:22,596 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,596 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1603): Closing 69a0168d9f604b96943e74817bde200a, disabling compactions & flushes 2023-02-14 15:59:22,596 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:22,596 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:22,596 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. after waiting 0 ms 2023-02-14 15:59:22,597 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:22,597 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:22,597 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1557): Region close journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:22,597 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(7671): creating {ENCODED => c064beb61994c28c74884e236a35d32b, NAME => 'testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676390362465', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp 2023-02-14 15:59:22,608 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,608 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1603): Closing 7ae37dd64d13c37bf459120525893ee5, disabling compactions & flushes 2023-02-14 15:59:22,608 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,609 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,609 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. after waiting 0 ms 2023-02-14 15:59:22,609 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,609 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,609 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-1] regionserver.HRegion(1557): Region close journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:22,612 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,613 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1603): Closing 7b10d6f25ee192486c2eccbdfd488bbb, disabling compactions & flushes 2023-02-14 15:59:22,613 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,613 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,613 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. after waiting 0 ms 2023-02-14 15:59:22,613 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,613 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,613 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-0] regionserver.HRegion(1557): Region close journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:22,620 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,620 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1603): Closing c064beb61994c28c74884e236a35d32b, disabling compactions & flushes 2023-02-14 15:59:22,621 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,621 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,621 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. after waiting 0 ms 2023-02-14 15:59:22,621 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,621 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,621 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676390362465-pool-2] regionserver.HRegion(1557): Region close journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:22,623 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 15:59:22,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362624"}]},"ts":"1676390362624"} 2023-02-14 15:59:22,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390362624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362624"}]},"ts":"1676390362624"} 2023-02-14 15:59:22,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362624"}]},"ts":"1676390362624"} 2023-02-14 15:59:22,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362624"}]},"ts":"1676390362624"} 2023-02-14 15:59:22,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362624"}]},"ts":"1676390362624"} 2023-02-14 15:59:22,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390362624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390362624"}]},"ts":"1676390362624"} 2023-02-14 15:59:22,630 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-14 15:59:22,633 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 15:59:22,633 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390362633"}]},"ts":"1676390362633"} 2023-02-14 15:59:22,635 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=ENABLING in hbase:meta 2023-02-14 15:59:22,641 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-14 15:59:22,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 15:59:22,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 15:59:22,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 15:59:22,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 15:59:22,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN}] 2023-02-14 15:59:22,647 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN 2023-02-14 15:59:22,647 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN 2023-02-14 15:59:22,647 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN 2023-02-14 15:59:22,647 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN 2023-02-14 15:59:22,648 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=false 2023-02-14 15:59:22,648 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=false 2023-02-14 15:59:22,648 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:59:22,648 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=false 2023-02-14 15:59:22,648 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN 2023-02-14 15:59:22,649 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN 2023-02-14 15:59:22,649 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=false 2023-02-14 15:59:22,649 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=false 2023-02-14 15:59:22,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-14 15:59:22,798 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:59:22,802 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:22,803 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:22,803 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:22,803 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362802"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390362802"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390362802"}]},"ts":"1676390362802"} 2023-02-14 15:59:22,803 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390362803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390362803"}]},"ts":"1676390362803"} 2023-02-14 15:59:22,802 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:22,803 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390362802"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390362802"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390362802"}]},"ts":"1676390362802"} 2023-02-14 15:59:22,803 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362802"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390362802"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390362802"}]},"ts":"1676390362802"} 2023-02-14 15:59:22,803 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:22,803 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390362803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390362803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390362803"}]},"ts":"1676390362803"} 2023-02-14 15:59:22,805 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=207, state=RUNNABLE; OpenRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:22,806 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=206, state=RUNNABLE; OpenRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:22,806 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=208, state=RUNNABLE; OpenRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:22,808 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:22,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=203, state=RUNNABLE; OpenRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:22,809 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:22,809 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390362809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390362809"}]},"ts":"1676390362809"} 2023-02-14 15:59:22,811 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=204, state=RUNNABLE; OpenRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:22,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7b10d6f25ee192486c2eccbdfd488bbb, NAME => 'testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:22,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7ae37dd64d13c37bf459120525893ee5, NAME => 'testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:22,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c064beb61994c28c74884e236a35d32b, NAME => 'testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,962 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,962 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,962 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,963 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf 2023-02-14 15:59:22,963 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf 2023-02-14 15:59:22,963 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf 2023-02-14 15:59:22,963 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf 2023-02-14 15:59:22,963 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7b10d6f25ee192486c2eccbdfd488bbb columnFamilyName cf 2023-02-14 15:59:22,964 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf 2023-02-14 15:59:22,964 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf 2023-02-14 15:59:22,964 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ae37dd64d13c37bf459120525893ee5 columnFamilyName cf 2023-02-14 15:59:22,964 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c064beb61994c28c74884e236a35d32b columnFamilyName cf 2023-02-14 15:59:22,965 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(310): Store=7ae37dd64d13c37bf459120525893ee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:22,965 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(310): Store=7b10d6f25ee192486c2eccbdfd488bbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:22,965 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(310): Store=c064beb61994c28c74884e236a35d32b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:22,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:22,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:22,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:22,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:59:22,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:59:22,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7ae37dd64d13c37bf459120525893ee5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68358885, jitterRate=0.01862676441669464}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:22,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:59:22,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:22,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c064beb61994c28c74884e236a35d32b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64495032, jitterRate=-0.03894913196563721}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:22,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:22,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7b10d6f25ee192486c2eccbdfd488bbb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67799062, jitterRate=0.010284751653671265}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:22,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:22,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5., pid=210, masterSystemTime=1676390362958 2023-02-14 15:59:22,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b., pid=211, masterSystemTime=1676390362959 2023-02-14 15:59:22,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb., pid=209, masterSystemTime=1676390362957 2023-02-14 15:59:22,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:22,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0d736737d4ae8d06714c55079f6c2b15, NAME => 'testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:22,974 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:22,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,974 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362974"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390362974"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390362974"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390362974"}]},"ts":"1676390362974"} 2023-02-14 15:59:22,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:22,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 69a0168d9f604b96943e74817bde200a, NAME => 'testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,976 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:22,976 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390362976"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390362976"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390362976"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390362976"}]},"ts":"1676390362976"} 2023-02-14 15:59:22,976 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:22,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:22,978 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:22,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 10c0bb63b1c83f26d2954d286d39b71f, NAME => 'testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:22,979 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390362978"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390362978"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390362978"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390362978"}]},"ts":"1676390362978"} 2023-02-14 15:59:22,979 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:22,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,979 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf 2023-02-14 15:59:22,979 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf 2023-02-14 15:59:22,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=208 2023-02-14 15:59:22,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=206 2023-02-14 15:59:22,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=206, state=SUCCESS; OpenRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146 in 170 msec 2023-02-14 15:59:22,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=208, state=SUCCESS; OpenRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069 in 171 msec 2023-02-14 15:59:22,980 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d736737d4ae8d06714c55079f6c2b15 columnFamilyName cf 2023-02-14 15:59:22,980 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf 2023-02-14 15:59:22,981 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf 2023-02-14 15:59:22,981 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,981 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 69a0168d9f604b96943e74817bde200a columnFamilyName cf 2023-02-14 15:59:22,981 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN in 337 msec 2023-02-14 15:59:22,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN in 337 msec 2023-02-14 15:59:22,981 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(310): Store=0d736737d4ae8d06714c55079f6c2b15/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:22,982 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(310): Store=69a0168d9f604b96943e74817bde200a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:22,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,982 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf 2023-02-14 15:59:22,982 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf 2023-02-14 15:59:22,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,982 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 10c0bb63b1c83f26d2954d286d39b71f columnFamilyName cf 2023-02-14 15:59:22,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,983 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(310): Store=10c0bb63b1c83f26d2954d286d39b71f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:22,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:22,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:22,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=207 2023-02-14 15:59:22,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=207, state=SUCCESS; OpenRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113 in 179 msec 2023-02-14 15:59:22,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:59:22,988 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN in 343 msec 2023-02-14 15:59:22,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0d736737d4ae8d06714c55079f6c2b15; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68594218, jitterRate=0.02213349938392639}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:22,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:22,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15., pid=214, masterSystemTime=1676390362958 2023-02-14 15:59:22,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:22,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:59:22,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:22,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 69a0168d9f604b96943e74817bde200a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72587328, jitterRate=0.0816354751586914}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:22,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:22,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 15:59:22,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a., pid=212, masterSystemTime=1676390362959 2023-02-14 15:59:22,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 10c0bb63b1c83f26d2954d286d39b71f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75491459, jitterRate=0.1249103993177414}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:22,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:22,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f., pid=213, masterSystemTime=1676390362957 2023-02-14 15:59:23,003 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,004 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363003"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390363003"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390363003"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390363003"}]},"ts":"1676390363003"} 2023-02-14 15:59:23,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:23,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:23,004 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,004 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363004"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390363004"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390363004"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390363004"}]},"ts":"1676390363004"} 2023-02-14 15:59:23,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,006 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,006 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363006"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390363006"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390363006"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390363006"}]},"ts":"1676390363006"} 2023-02-14 15:59:23,008 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=204 2023-02-14 15:59:23,008 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=204, state=SUCCESS; OpenRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146 in 194 msec 2023-02-14 15:59:23,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-14 15:59:23,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069 in 198 msec 2023-02-14 15:59:23,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN in 365 msec 2023-02-14 15:59:23,010 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=203 2023-02-14 15:59:23,010 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=203, state=SUCCESS; OpenRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113 in 199 msec 2023-02-14 15:59:23,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN in 366 msec 2023-02-14 15:59:23,012 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=203, resume processing ppid=202 2023-02-14 15:59:23,012 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN in 367 msec 2023-02-14 15:59:23,013 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 15:59:23,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390363013"}]},"ts":"1676390363013"} 2023-02-14 15:59:23,015 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=ENABLED in hbase:meta 2023-02-14 15:59:23,021 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 15:59:23,023 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676390362465 in 553 msec 2023-02-14 15:59:23,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-14 15:59:23,226 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 202 completed 2023-02-14 15:59:23,229 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,229 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,232 INFO [Listener at localhost/39279] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 15:59:23,246 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,247 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-14 15:59:23,252 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390363252"}]},"ts":"1676390363252"} 2023-02-14 15:59:23,255 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=DISABLING in hbase:meta 2023-02-14 15:59:23,257 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676390362465 to state=DISABLING 2023-02-14 15:59:23,259 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN}] 2023-02-14 15:59:23,260 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN 2023-02-14 15:59:23,260 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN 2023-02-14 15:59:23,260 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN 2023-02-14 15:59:23,260 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN 2023-02-14 15:59:23,260 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN 2023-02-14 15:59:23,261 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,261 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363261"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363261"}]},"ts":"1676390363261"} 2023-02-14 15:59:23,262 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,262 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363261"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363261"}]},"ts":"1676390363261"} 2023-02-14 15:59:23,262 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,262 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363262"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363262"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363262"}]},"ts":"1676390363262"} 2023-02-14 15:59:23,262 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,262 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,262 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363262"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363262"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363262"}]},"ts":"1676390363262"} 2023-02-14 15:59:23,262 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363262"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363262"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363262"}]},"ts":"1676390363262"} 2023-02-14 15:59:23,264 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=221, state=RUNNABLE; CloseRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:23,265 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=220, state=RUNNABLE; CloseRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:23,266 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=219, state=RUNNABLE; CloseRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:23,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=217, state=RUNNABLE; CloseRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:23,269 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=218, state=RUNNABLE; CloseRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:23,270 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN 2023-02-14 15:59:23,270 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,271 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363270"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363270"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363270"}]},"ts":"1676390363270"} 2023-02-14 15:59:23,272 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:23,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:23,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c064beb61994c28c74884e236a35d32b, disabling compactions & flushes 2023-02-14 15:59:23,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:23,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:23,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. after waiting 0 ms 2023-02-14 15:59:23,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:23,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:23,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7b10d6f25ee192486c2eccbdfd488bbb, disabling compactions & flushes 2023-02-14 15:59:23,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:23,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:23,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. after waiting 0 ms 2023-02-14 15:59:23,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:23,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:23,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0d736737d4ae8d06714c55079f6c2b15, disabling compactions & flushes 2023-02-14 15:59:23,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:23,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:23,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. after waiting 0 ms 2023-02-14 15:59:23,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:23,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:59:23,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:23,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:23,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:59:23,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:23,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:23,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:23,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:23,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 10c0bb63b1c83f26d2954d286d39b71f, disabling compactions & flushes 2023-02-14 15:59:23,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. after waiting 0 ms 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,424 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=CLOSED 2023-02-14 15:59:23,424 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363424"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390363424"}]},"ts":"1676390363424"} 2023-02-14 15:59:23,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:23,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 69a0168d9f604b96943e74817bde200a, disabling compactions & flushes 2023-02-14 15:59:23,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. after waiting 0 ms 2023-02-14 15:59:23,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:23,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:59:23,425 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=CLOSED 2023-02-14 15:59:23,425 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363425"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390363425"}]},"ts":"1676390363425"} 2023-02-14 15:59:23,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:23,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:23,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:23,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:23,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7ae37dd64d13c37bf459120525893ee5, disabling compactions & flushes 2023-02-14 15:59:23,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:23,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:23,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. after waiting 0 ms 2023-02-14 15:59:23,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:23,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:59:23,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:23,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:23,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:59:23,429 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=CLOSED 2023-02-14 15:59:23,429 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363429"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390363429"}]},"ts":"1676390363429"} 2023-02-14 15:59:23,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:23,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=220 2023-02-14 15:59:23,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=221 2023-02-14 15:59:23,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=221, state=SUCCESS; CloseRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069 in 163 msec 2023-02-14 15:59:23,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:23,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:23,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=220, state=SUCCESS; CloseRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113 in 161 msec 2023-02-14 15:59:23,431 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=CLOSED 2023-02-14 15:59:23,431 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363431"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390363431"}]},"ts":"1676390363431"} 2023-02-14 15:59:23,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN in 172 msec 2023-02-14 15:59:23,432 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN in 173 msec 2023-02-14 15:59:23,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:23,433 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=CLOSED 2023-02-14 15:59:23,433 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390363433"}]},"ts":"1676390363433"} 2023-02-14 15:59:23,434 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-14 15:59:23,434 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113 in 161 msec 2023-02-14 15:59:23,436 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=217 2023-02-14 15:59:23,436 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=217, state=SUCCESS; CloseRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146 in 166 msec 2023-02-14 15:59:23,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 15:59:23,436 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN in 176 msec 2023-02-14 15:59:23,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=218 2023-02-14 15:59:23,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=218, state=SUCCESS; CloseRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069 in 165 msec 2023-02-14 15:59:23,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:23,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:23,437 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN in 178 msec 2023-02-14 15:59:23,437 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN in 178 msec 2023-02-14 15:59:23,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:23,438 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=CLOSED 2023-02-14 15:59:23,438 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363438"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390363438"}]},"ts":"1676390363438"} 2023-02-14 15:59:23,441 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=219 2023-02-14 15:59:23,441 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=219, state=SUCCESS; CloseRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146 in 173 msec 2023-02-14 15:59:23,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=219, resume processing ppid=215 2023-02-14 15:59:23,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN in 183 msec 2023-02-14 15:59:23,442 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390363442"}]},"ts":"1676390363442"} 2023-02-14 15:59:23,443 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=DISABLED in hbase:meta 2023-02-14 15:59:23,445 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676390362465 to state=DISABLED 2023-02-14 15:59:23,448 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676390362465 in 200 msec 2023-02-14 15:59:23,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-14 15:59:23,502 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 215 completed 2023-02-14 15:59:23,503 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } 2023-02-14 15:59:23,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390363503 (current time:1676390363503). 2023-02-14 15:59:23,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:59:23,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x06c6a9f8 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:23,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4270c910, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:23,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:23,512 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33312, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06c6a9f8 to 127.0.0.1:55782 2023-02-14 15:59:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:59:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } 2023-02-14 15:59:23,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE 2023-02-14 15:59:23,516 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE 2023-02-14 15:59:23,517 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE; timeout=600000, timestamp=1676390963517 2023-02-14 15:59:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } 2023-02-14 15:59:23,517 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676390362465 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,517 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:59:23,518 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:59:23,542 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=DISABLED ttl=0 } 2023-02-14 15:59:23,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,549 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,549 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,550 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,551 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:23,551 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:23,551 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,551 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,559 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,560 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,560 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,562 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-14 15:59:23,608 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=DISABLED ttl=0 } as finished. 2023-02-14 15:59:23,608 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676390362465 2023-02-14 15:59:23,609 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390362465 2023-02-14 15:59:23,639 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390362465 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/emptySnaptb-1676390362465 2023-02-14 15:59:23,655 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676390362465 of table testCorruptedSnapshot-1676390362465 completed 2023-02-14 15:59:23,656 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390362465 2023-02-14 15:59:23,656 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/emptySnaptb-1676390362465 2023-02-14 15:59:23,656 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676390362465 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676390362465 at 1676390363517Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=DISABLED ttl=0 } at 1676390363542 (+25 ms)Consolidate snapshot: emptySnaptb-1676390362465 at 1676390363608 (+66 ms)Loading Region manifests for emptySnaptb-1676390362465 at 1676390363608Writing data manifest for emptySnaptb-1676390362465 at 1676390363617 (+9 ms)Verifying snapshot: emptySnaptb-1676390362465 at 1676390363634 (+17 ms)Snapshot emptySnaptb-1676390362465 of table testCorruptedSnapshot-1676390362465 completed at 1676390363655 (+21 ms) 2023-02-14 15:59:23,657 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE 2023-02-14 15:59:23,658 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE in 142 msec 2023-02-14 15:59:23,768 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:23,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } is done 2023-02-14 15:59:23,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:59:23,769 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,770 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-14 15:59:23,773 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:23,773 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390363773"}]},"ts":"1676390363773"} 2023-02-14 15:59:23,775 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=ENABLING in hbase:meta 2023-02-14 15:59:23,777 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:59:23,777 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN}] 2023-02-14 15:59:23,778 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN 2023-02-14 15:59:23,778 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN 2023-02-14 15:59:23,779 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN 2023-02-14 15:59:23,779 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN 2023-02-14 15:59:23,779 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,779 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,779 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:59:23,779 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:59:23,779 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN 2023-02-14 15:59:23,780 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,780 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,780 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:59:23,780 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:59:23,780 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN 2023-02-14 15:59:23,780 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,781 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:59:23,781 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,781 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:59:23,930 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:59:23,930 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,930 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,930 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,930 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:23,930 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363930"}]},"ts":"1676390363930"} 2023-02-14 15:59:23,930 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363930"}]},"ts":"1676390363930"} 2023-02-14 15:59:23,930 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:23,930 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363930"}]},"ts":"1676390363930"} 2023-02-14 15:59:23,931 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390363930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363930"}]},"ts":"1676390363930"} 2023-02-14 15:59:23,930 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363930"}]},"ts":"1676390363930"} 2023-02-14 15:59:23,932 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=232, state=RUNNABLE; OpenRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:23,932 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=230, state=RUNNABLE; OpenRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:23,933 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=234, state=RUNNABLE; OpenRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:23,934 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=235, state=RUNNABLE; OpenRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:23,935 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=233, state=RUNNABLE; OpenRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:23,935 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:23,935 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390363935"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390363935"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390363935"}]},"ts":"1676390363935"} 2023-02-14 15:59:23,937 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=231, state=RUNNABLE; OpenRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-14 15:59:24,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:24,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7b10d6f25ee192486c2eccbdfd488bbb, NAME => 'testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c064beb61994c28c74884e236a35d32b, NAME => 'testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,088 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,089 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,090 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf 2023-02-14 15:59:24,090 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf 2023-02-14 15:59:24,090 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf 2023-02-14 15:59:24,090 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c064beb61994c28c74884e236a35d32b columnFamilyName cf 2023-02-14 15:59:24,090 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf 2023-02-14 15:59:24,091 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7b10d6f25ee192486c2eccbdfd488bbb columnFamilyName cf 2023-02-14 15:59:24,091 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(310): Store=c064beb61994c28c74884e236a35d32b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:24,091 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0d736737d4ae8d06714c55079f6c2b15, NAME => 'testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:24,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:24,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,092 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(310): Store=7b10d6f25ee192486c2eccbdfd488bbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:24,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,093 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,094 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf 2023-02-14 15:59:24,094 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf 2023-02-14 15:59:24,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,094 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d736737d4ae8d06714c55079f6c2b15 columnFamilyName cf 2023-02-14 15:59:24,095 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(310): Store=0d736737d4ae8d06714c55079f6c2b15/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:24,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,103 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c064beb61994c28c74884e236a35d32b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72430323, jitterRate=0.07929591834545135}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:24,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:24,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7b10d6f25ee192486c2eccbdfd488bbb; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61363539, jitterRate=-0.08561201393604279}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:24,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:24,105 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb., pid=238, masterSystemTime=1676390364084 2023-02-14 15:59:24,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 10c0bb63b1c83f26d2954d286d39b71f, NAME => 'testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:24,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:24,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,107 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:24,108 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364107"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390364107"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390364107"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390364107"}]},"ts":"1676390364107"} 2023-02-14 15:59:24,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b., pid=239, masterSystemTime=1676390364084 2023-02-14 15:59:24,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0d736737d4ae8d06714c55079f6c2b15; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65329481, jitterRate=-0.026514872908592224}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:24,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:24,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15., pid=241, masterSystemTime=1676390364087 2023-02-14 15:59:24,111 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:24,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:24,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 69a0168d9f604b96943e74817bde200a, NAME => 'testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,112 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:24,112 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390364112"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390364112"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390364112"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390364112"}]},"ts":"1676390364112"} 2023-02-14 15:59:24,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7ae37dd64d13c37bf459120525893ee5, NAME => 'testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:24,113 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf 2023-02-14 15:59:24,113 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf 2023-02-14 15:59:24,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:24,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,113 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 10c0bb63b1c83f26d2954d286d39b71f columnFamilyName cf 2023-02-14 15:59:24,114 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:24,114 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364114"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390364114"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390364114"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390364114"}]},"ts":"1676390364114"} 2023-02-14 15:59:24,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=234 2023-02-14 15:59:24,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=234, state=SUCCESS; OpenRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113 in 177 msec 2023-02-14 15:59:24,114 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,115 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(310): Store=10c0bb63b1c83f26d2954d286d39b71f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:24,115 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN in 337 msec 2023-02-14 15:59:24,115 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf 2023-02-14 15:59:24,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,116 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,116 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf 2023-02-14 15:59:24,117 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 69a0168d9f604b96943e74817bde200a columnFamilyName cf 2023-02-14 15:59:24,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=235 2023-02-14 15:59:24,118 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf 2023-02-14 15:59:24,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=235, state=SUCCESS; OpenRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069 in 180 msec 2023-02-14 15:59:24,118 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf 2023-02-14 15:59:24,118 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(310): Store=69a0168d9f604b96943e74817bde200a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:24,118 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ae37dd64d13c37bf459120525893ee5 columnFamilyName cf 2023-02-14 15:59:24,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,119 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=231 2023-02-14 15:59:24,119 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=231, state=SUCCESS; OpenRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146 in 178 msec 2023-02-14 15:59:24,119 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(310): Store=7ae37dd64d13c37bf459120525893ee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:24,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN in 341 msec 2023-02-14 15:59:24,120 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN in 342 msec 2023-02-14 15:59:24,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 10c0bb63b1c83f26d2954d286d39b71f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70618766, jitterRate=0.052301615476608276}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:24,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:24,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f., pid=237, masterSystemTime=1676390364084 2023-02-14 15:59:24,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:24,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 69a0168d9f604b96943e74817bde200a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73349359, jitterRate=0.09299062192440033}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:24,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:24,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,125 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:24,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a., pid=236, masterSystemTime=1676390364084 2023-02-14 15:59:24,125 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390364125"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390364125"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390364125"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390364125"}]},"ts":"1676390364125"} 2023-02-14 15:59:24,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7ae37dd64d13c37bf459120525893ee5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69784830, jitterRate=0.03987500071525574}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:24,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:24,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5., pid=240, masterSystemTime=1676390364087 2023-02-14 15:59:24,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:24,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:24,127 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:24,127 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364127"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390364127"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390364127"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390364127"}]},"ts":"1676390364127"} 2023-02-14 15:59:24,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,128 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:24,128 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364128"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390364128"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390364128"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390364128"}]},"ts":"1676390364128"} 2023-02-14 15:59:24,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=230 2023-02-14 15:59:24,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=230, state=SUCCESS; OpenRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113 in 194 msec 2023-02-14 15:59:24,131 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=232 2023-02-14 15:59:24,131 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=232, state=SUCCESS; OpenRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069 in 197 msec 2023-02-14 15:59:24,131 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN in 352 msec 2023-02-14 15:59:24,132 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=233 2023-02-14 15:59:24,132 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN in 354 msec 2023-02-14 15:59:24,132 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=233, state=SUCCESS; OpenRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146 in 194 msec 2023-02-14 15:59:24,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=233, resume processing ppid=229 2023-02-14 15:59:24,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN in 355 msec 2023-02-14 15:59:24,134 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390364134"}]},"ts":"1676390364134"} 2023-02-14 15:59:24,136 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=ENABLED in hbase:meta 2023-02-14 15:59:24,137 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676390362465' was successfully enabled. 2023-02-14 15:59:24,140 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676390362465 in 368 msec 2023-02-14 15:59:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-14 15:59:24,525 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 229 completed 2023-02-14 15:59:24,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:24,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:24,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:24,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:24,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:24,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:24,539 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:24,539 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,540 INFO [Listener at localhost/39279] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 15:59:24,552 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:24,553 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:24,554 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:24,555 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:24,557 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:24,560 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676390362465 2023-02-14 15:59:24,560 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676390362465 2023-02-14 15:59:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:24,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-14 15:59:24,564 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390364564"}]},"ts":"1676390364564"} 2023-02-14 15:59:24,566 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=DISABLING in hbase:meta 2023-02-14 15:59:24,568 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676390362465 to state=DISABLING 2023-02-14 15:59:24,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN}] 2023-02-14 15:59:24,570 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN 2023-02-14 15:59:24,570 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN 2023-02-14 15:59:24,570 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN 2023-02-14 15:59:24,571 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:24,571 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:24,571 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:24,571 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390364571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390364571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390364571"}]},"ts":"1676390364571"} 2023-02-14 15:59:24,571 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390364571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390364571"}]},"ts":"1676390364571"} 2023-02-14 15:59:24,571 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390364571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390364571"}]},"ts":"1676390364571"} 2023-02-14 15:59:24,571 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN 2023-02-14 15:59:24,572 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN 2023-02-14 15:59:24,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=243, state=RUNNABLE; CloseRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:24,573 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:24,573 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364573"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390364573"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390364573"}]},"ts":"1676390364573"} 2023-02-14 15:59:24,573 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:24,573 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390364573"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390364573"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390364573"}]},"ts":"1676390364573"} 2023-02-14 15:59:24,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=246, state=RUNNABLE; CloseRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:24,574 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:24,575 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=245, state=RUNNABLE; CloseRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:24,576 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=248, state=RUNNABLE; CloseRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:24,576 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN 2023-02-14 15:59:24,577 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:24,577 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364577"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390364577"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390364577"}]},"ts":"1676390364577"} 2023-02-14 15:59:24,579 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:24,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 10c0bb63b1c83f26d2954d286d39b71f, disabling compactions & flushes 2023-02-14 15:59:24,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. after waiting 0 ms 2023-02-14 15:59:24,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 10c0bb63b1c83f26d2954d286d39b71f 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-14 15:59:24,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0d736737d4ae8d06714c55079f6c2b15, disabling compactions & flushes 2023-02-14 15:59:24,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. after waiting 0 ms 2023-02-14 15:59:24,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0d736737d4ae8d06714c55079f6c2b15 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-14 15:59:24,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:24,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c064beb61994c28c74884e236a35d32b, disabling compactions & flushes 2023-02-14 15:59:24,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:24,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:24,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. after waiting 0 ms 2023-02-14 15:59:24,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:24,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c064beb61994c28c74884e236a35d32b 1/1 column families, dataSize=22.37 KB heapSize=48.34 KB 2023-02-14 15:59:24,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230214f2d257aa345040b4818473289f6801fa to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b20230214f2d257aa345040b4818473289f6801fa 2023-02-14 15:59:24,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302146b061d5d61164507a29090bb79b300ad to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e202302146b061d5d61164507a29090bb79b300ad 2023-02-14 15:59:24,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230214ebb5b31f9c904432a43804d8daf83134 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214ebb5b31f9c904432a43804d8daf83134 2023-02-14 15:59:24,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/.tmp/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a 2023-02-14 15:59:24,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/.tmp/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a 2023-02-14 15:59:24,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a, entries=30, sequenceid=8, filesize=10.1 K 2023-02-14 15:59:24,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 0d736737d4ae8d06714c55079f6c2b15 in 83ms, sequenceid=8, compaction requested=false 2023-02-14 15:59:24,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676390362465' 2023-02-14 15:59:24,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/.tmp/cf/41125881fafa49fbb0fe6c39d7da77fb 2023-02-14 15:59:24,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-14 15:59:24,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/.tmp/cf/41125881fafa49fbb0fe6c39d7da77fb as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb 2023-02-14 15:59:24,820 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb, entries=31, sequenceid=8, filesize=10.2 K 2023-02-14 15:59:24,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 10c0bb63b1c83f26d2954d286d39b71f in 96ms, sequenceid=8, compaction requested=false 2023-02-14 15:59:24,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:59:24,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:59:24,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:24,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:24,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:24,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:24,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:24,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7ae37dd64d13c37bf459120525893ee5, disabling compactions & flushes 2023-02-14 15:59:24,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. after waiting 0 ms 2023-02-14 15:59:24,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,830 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=CLOSED 2023-02-14 15:59:24,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7ae37dd64d13c37bf459120525893ee5 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-14 15:59:24,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:24,831 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364830"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390364830"}]},"ts":"1676390364830"} 2023-02-14 15:59:24,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7b10d6f25ee192486c2eccbdfd488bbb, disabling compactions & flushes 2023-02-14 15:59:24,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. after waiting 0 ms 2023-02-14 15:59:24,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7b10d6f25ee192486c2eccbdfd488bbb 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-14 15:59:24,831 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=CLOSED 2023-02-14 15:59:24,831 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390364831"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390364831"}]},"ts":"1676390364831"} 2023-02-14 15:59:24,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-02-14 15:59:24,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146 in 262 msec 2023-02-14 15:59:24,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=243 2023-02-14 15:59:24,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=243, state=SUCCESS; CloseRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113 in 265 msec 2023-02-14 15:59:24,839 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN in 269 msec 2023-02-14 15:59:24,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN in 269 msec 2023-02-14 15:59:24,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230214628c28ba880c473db782fae69e0c437b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214628c28ba880c473db782fae69e0c437b 2023-02-14 15:59:24,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/.tmp/cf/55a86829b0f24b879d7ddccf917a6091 2023-02-14 15:59:24,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302143ccf38443fb144958318754b8a6f8693 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c202302143ccf38443fb144958318754b8a6f8693 2023-02-14 15:59:24,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/.tmp/cf/55a86829b0f24b879d7ddccf917a6091 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091 2023-02-14 15:59:24,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091, entries=25, sequenceid=8, filesize=9.2 K 2023-02-14 15:59:24,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 7ae37dd64d13c37bf459120525893ee5 in 55ms, sequenceid=8, compaction requested=false 2023-02-14 15:59:24,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/.tmp/cf/319d737998834da7ad9f99a4683cfdc2 2023-02-14 15:59:24,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:59:24,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:24,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:24,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:24,901 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=CLOSED 2023-02-14 15:59:24,902 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390364901"}]},"ts":"1676390364901"} 2023-02-14 15:59:24,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/.tmp/cf/319d737998834da7ad9f99a4683cfdc2 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2 2023-02-14 15:59:24,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=246 2023-02-14 15:59:24,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=246, state=SUCCESS; CloseRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146 in 330 msec 2023-02-14 15:59:24,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2, entries=36, sequenceid=8, filesize=11.1 K 2023-02-14 15:59:24,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 7b10d6f25ee192486c2eccbdfd488bbb in 76ms, sequenceid=8, compaction requested=false 2023-02-14 15:59:24,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN in 335 msec 2023-02-14 15:59:24,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:59:24,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:24,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:24,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:24,913 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=CLOSED 2023-02-14 15:59:24,913 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390364913"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390364913"}]},"ts":"1676390364913"} 2023-02-14 15:59:24,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-14 15:59:24,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113 in 335 msec 2023-02-14 15:59:24,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN in 351 msec 2023-02-14 15:59:25,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/.tmp/cf/e0e4bbcdc1a04531af790ce69c79563e 2023-02-14 15:59:25,229 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/.tmp/cf/e0e4bbcdc1a04531af790ce69c79563e as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e 2023-02-14 15:59:25,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e, entries=342, sequenceid=8, filesize=63.8 K 2023-02-14 15:59:25,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.37 KB/22910, heapSize ~48.33 KB/49488, currentSize=0 B/0 for c064beb61994c28c74884e236a35d32b in 505ms, sequenceid=8, compaction requested=false 2023-02-14 15:59:25,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:59:25,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:25,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:25,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:25,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:25,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 69a0168d9f604b96943e74817bde200a, disabling compactions & flushes 2023-02-14 15:59:25,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:25,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:25,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. after waiting 0 ms 2023-02-14 15:59:25,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:25,242 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=CLOSED 2023-02-14 15:59:25,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 69a0168d9f604b96943e74817bde200a 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-14 15:59:25,242 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390365242"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390365242"}]},"ts":"1676390365242"} 2023-02-14 15:59:25,245 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=248 2023-02-14 15:59:25,245 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=248, state=SUCCESS; CloseRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069 in 667 msec 2023-02-14 15:59:25,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN in 676 msec 2023-02-14 15:59:25,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230214871a60efa2e340b590b0521d8fc5ca5f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214871a60efa2e340b590b0521d8fc5ca5f 2023-02-14 15:59:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-14 15:59:25,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/.tmp/cf/c6cef7dc49e2452f8c44360784c16493 2023-02-14 15:59:25,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/.tmp/cf/c6cef7dc49e2452f8c44360784c16493 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493 2023-02-14 15:59:25,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493, entries=36, sequenceid=8, filesize=11.1 K 2023-02-14 15:59:25,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 69a0168d9f604b96943e74817bde200a in 443ms, sequenceid=8, compaction requested=false 2023-02-14 15:59:25,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-14 15:59:25,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:25,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:25,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:25,700 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=CLOSED 2023-02-14 15:59:25,700 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390365700"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390365700"}]},"ts":"1676390365700"} 2023-02-14 15:59:25,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=245 2023-02-14 15:59:25,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=245, state=SUCCESS; CloseRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069 in 1.1270 sec 2023-02-14 15:59:25,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=245, resume processing ppid=242 2023-02-14 15:59:25,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN in 1.1370 sec 2023-02-14 15:59:25,710 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390365710"}]},"ts":"1676390365710"} 2023-02-14 15:59:25,711 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=DISABLED in hbase:meta 2023-02-14 15:59:25,714 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676390362465 to state=DISABLED 2023-02-14 15:59:25,717 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676390362465 in 1.1550 sec 2023-02-14 15:59:26,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-14 15:59:26,069 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 242 completed 2023-02-14 15:59:26,069 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } 2023-02-14 15:59:26,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676390366070 (current time:1676390366070). 2023-02-14 15:59:26,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-14 15:59:26,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(139): Connect 0x4091dc25 to 127.0.0.1:55782 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 15:59:26,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@732b34a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 15:59:26,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 15:59:26,086 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33324, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 15:59:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4091dc25 to 127.0.0.1:55782 2023-02-14 15:59:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-14 15:59:26,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } 2023-02-14 15:59:26,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE 2023-02-14 15:59:26,090 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE 2023-02-14 15:59:26,091 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE; timeout=600000, timestamp=1676390966091 2023-02-14 15:59:26,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } 2023-02-14 15:59:26,091 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676390362465 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:26,091 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-14 15:59:26,091 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-14 15:59:26,100 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=DISABLED ttl=0 } 2023-02-14 15:59:26,102 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,102 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,102 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,102 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,102 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,102 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,103 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,103 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,103 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,103 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,103 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,104 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,104 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-14 15:59:26,104 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-14 15:59:26,105 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091] hfiles 2023-02-14 15:59:26,105 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091 2023-02-14 15:59:26,105 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493] hfiles 2023-02-14 15:59:26,105 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2] hfiles 2023-02-14 15:59:26,105 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e] hfiles 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb] hfiles 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a] hfiles 2023-02-14 15:59:26,106 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a 2023-02-14 15:59:26,107 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c202302143ccf38443fb144958318754b8a6f8693, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b20230214f2d257aa345040b4818473289f6801fa, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214871a60efa2e340b590b0521d8fc5ca5f, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e202302146b061d5d61164507a29090bb79b300ad, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214ebb5b31f9c904432a43804d8daf83134, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214628c28ba880c473db782fae69e0c437b] hfiles 2023-02-14 15:59:26,107 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c202302143ccf38443fb144958318754b8a6f8693 2023-02-14 15:59:26,107 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b20230214f2d257aa345040b4818473289f6801fa 2023-02-14 15:59:26,108 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214871a60efa2e340b590b0521d8fc5ca5f 2023-02-14 15:59:26,108 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e202302146b061d5d61164507a29090bb79b300ad 2023-02-14 15:59:26,108 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214ebb5b31f9c904432a43804d8daf83134 2023-02-14 15:59:26,108 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214628c28ba880c473db782fae69e0c437b 2023-02-14 15:59:26,126 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=DISABLED ttl=0 } as finished. 2023-02-14 15:59:26,126 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676390362465 2023-02-14 15:59:26,127 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390362465 2023-02-14 15:59:26,143 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390362465 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/snaptb0-1676390362465 2023-02-14 15:59:26,161 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676390362465 of table testCorruptedSnapshot-1676390362465 completed 2023-02-14 15:59:26,161 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390362465 2023-02-14 15:59:26,161 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.hbase-snapshot/.tmp/snaptb0-1676390362465 2023-02-14 15:59:26,161 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676390362465 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676390362465 at 1676390366091Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=DISABLED ttl=0 } at 1676390366100 (+9 ms)Consolidate snapshot: snaptb0-1676390362465 at 1676390366126 (+26 ms)Loading Region manifests for snaptb0-1676390362465 at 1676390366126Writing data manifest for snaptb0-1676390362465 at 1676390366129 (+3 ms)Verifying snapshot: snaptb0-1676390362465 at 1676390366137 (+8 ms)Snapshot snaptb0-1676390362465 of table testCorruptedSnapshot-1676390362465 completed at 1676390366161 (+24 ms) 2023-02-14 15:59:26,162 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE 2023-02-14 15:59:26,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676390362465, type=EXCLUSIVE in 73 msec 2023-02-14 15:59:26,342 DEBUG [Listener at localhost/39279] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-14 15:59:26,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 } is done 2023-02-14 15:59:26,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676390362465 table=testCorruptedSnapshot-1676390362465 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-14 15:59:26,343 INFO [Listener at localhost/39279] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676390362465 2023-02-14 15:59:26,343 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676390362465 2023-02-14 15:59:26,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:26,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-14 15:59:26,347 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:26,347 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390366347"}]},"ts":"1676390366347"} 2023-02-14 15:59:26,348 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=ENABLING in hbase:meta 2023-02-14 15:59:26,350 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-14 15:59:26,350 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN}] 2023-02-14 15:59:26,350 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN 2023-02-14 15:59:26,350 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN 2023-02-14 15:59:26,351 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN 2023-02-14 15:59:26,351 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN 2023-02-14 15:59:26,351 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN 2023-02-14 15:59:26,351 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:26,351 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:26,351 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:26,351 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:59:26,351 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:59:26,351 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:26,351 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43707,1676390331069; forceNewPlan=false, retain=true 2023-02-14 15:59:26,352 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:59:26,352 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:26,352 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37877,1676390331146; forceNewPlan=false, retain=true 2023-02-14 15:59:26,352 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN 2023-02-14 15:59:26,352 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:26,352 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40531,1676390331113; forceNewPlan=false, retain=true 2023-02-14 15:59:26,502 INFO [jenkins-hbase4:33899] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-14 15:59:26,502 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:26,502 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:26,502 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390366502"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390366502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390366502"}]},"ts":"1676390366502"} 2023-02-14 15:59:26,502 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:26,502 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:26,503 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366502"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390366502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390366502"}]},"ts":"1676390366502"} 2023-02-14 15:59:26,502 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:26,503 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390366502"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390366502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390366502"}]},"ts":"1676390366502"} 2023-02-14 15:59:26,502 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366502"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390366502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390366502"}]},"ts":"1676390366502"} 2023-02-14 15:59:26,502 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366502"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390366502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390366502"}]},"ts":"1676390366502"} 2023-02-14 15:59:26,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:26,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=261, state=RUNNABLE; OpenRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:26,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=262, state=RUNNABLE; OpenRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:26,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:26,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=260, state=RUNNABLE; OpenRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:26,510 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:26,510 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390366509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390366509"}]},"ts":"1676390366509"} 2023-02-14 15:59:26,511 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=258, state=RUNNABLE; OpenRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:26,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-14 15:59:26,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:26,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7b10d6f25ee192486c2eccbdfd488bbb, NAME => 'testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-14 15:59:26,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:26,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,661 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:26,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c064beb61994c28c74884e236a35d32b, NAME => 'testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.', STARTKEY => '5', ENDKEY => ''} 2023-02-14 15:59:26,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:26,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,663 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf 2023-02-14 15:59:26,663 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf 2023-02-14 15:59:26,664 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7b10d6f25ee192486c2eccbdfd488bbb columnFamilyName cf 2023-02-14 15:59:26,664 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:26,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7ae37dd64d13c37bf459120525893ee5, NAME => 'testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.', STARTKEY => '3', ENDKEY => '4'} 2023-02-14 15:59:26,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:26,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,665 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf 2023-02-14 15:59:26,665 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf 2023-02-14 15:59:26,665 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c064beb61994c28c74884e236a35d32b columnFamilyName cf 2023-02-14 15:59:26,666 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,667 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf 2023-02-14 15:59:26,667 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf 2023-02-14 15:59:26,667 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ae37dd64d13c37bf459120525893ee5 columnFamilyName cf 2023-02-14 15:59:26,669 DEBUG [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2 2023-02-14 15:59:26,671 INFO [StoreOpener-7b10d6f25ee192486c2eccbdfd488bbb-1] regionserver.HStore(310): Store=7b10d6f25ee192486c2eccbdfd488bbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:26,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,676 DEBUG [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091 2023-02-14 15:59:26,676 INFO [StoreOpener-7ae37dd64d13c37bf459120525893ee5-1] regionserver.HStore(310): Store=7ae37dd64d13c37bf459120525893ee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:26,676 DEBUG [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e 2023-02-14 15:59:26,676 INFO [StoreOpener-c064beb61994c28c74884e236a35d32b-1] regionserver.HStore(310): Store=c064beb61994c28c74884e236a35d32b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:26,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:26,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7b10d6f25ee192486c2eccbdfd488bbb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61325680, jitterRate=-0.08617615699768066}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:26,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:26,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb., pid=264, masterSystemTime=1676390366657 2023-02-14 15:59:26,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:26,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:26,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 10c0bb63b1c83f26d2954d286d39b71f, NAME => 'testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.', STARTKEY => '', ENDKEY => '1'} 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7ae37dd64d13c37bf459120525893ee5; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61376720, jitterRate=-0.08541560173034668}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:26,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:26,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c064beb61994c28c74884e236a35d32b; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65269203, jitterRate=-0.027413085103034973}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:26,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:26,684 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:26,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5., pid=267, masterSystemTime=1676390366661 2023-02-14 15:59:26,684 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366684"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390366684"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390366684"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390366684"}]},"ts":"1676390366684"} 2023-02-14 15:59:26,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b., pid=265, masterSystemTime=1676390366660 2023-02-14 15:59:26,685 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:26,687 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf 2023-02-14 15:59:26,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:26,687 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf 2023-02-14 15:59:26,687 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:26,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0d736737d4ae8d06714c55079f6c2b15, NAME => 'testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.', STARTKEY => '1', ENDKEY => '2'} 2023-02-14 15:59:26,687 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 10c0bb63b1c83f26d2954d286d39b71f columnFamilyName cf 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,687 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366687"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390366687"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390366687"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390366687"}]},"ts":"1676390366687"} 2023-02-14 15:59:26,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:26,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:26,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 69a0168d9f604b96943e74817bde200a, NAME => 'testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.', STARTKEY => '2', ENDKEY => '3'} 2023-02-14 15:59:26,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676390362465 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 15:59:26,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,690 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,692 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf 2023-02-14 15:59:26,692 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf 2023-02-14 15:59:26,692 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 69a0168d9f604b96943e74817bde200a columnFamilyName cf 2023-02-14 15:59:26,692 DEBUG [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb 2023-02-14 15:59:26,693 INFO [StoreOpener-10c0bb63b1c83f26d2954d286d39b71f-1] regionserver.HStore(310): Store=10c0bb63b1c83f26d2954d286d39b71f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:26,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,696 DEBUG [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493 2023-02-14 15:59:26,697 INFO [StoreOpener-69a0168d9f604b96943e74817bde200a-1] regionserver.HStore(310): Store=69a0168d9f604b96943e74817bde200a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:26,697 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:26,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=261 2023-02-14 15:59:26,697 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390366697"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390366697"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390366697"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390366697"}]},"ts":"1676390366697"} 2023-02-14 15:59:26,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=261, state=SUCCESS; OpenRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113 in 180 msec 2023-02-14 15:59:26,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:26,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=260 2023-02-14 15:59:26,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=260, state=SUCCESS; OpenRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146 in 181 msec 2023-02-14 15:59:26,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 10c0bb63b1c83f26d2954d286d39b71f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61231673, jitterRate=-0.0875769704580307}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:26,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:26,699 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, ASSIGN in 347 msec 2023-02-14 15:59:26,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f., pid=263, masterSystemTime=1676390366657 2023-02-14 15:59:26,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, ASSIGN in 348 msec 2023-02-14 15:59:26,700 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=262 2023-02-14 15:59:26,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=262, state=SUCCESS; OpenRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069 in 191 msec 2023-02-14 15:59:26,701 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf 2023-02-14 15:59:26,701 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf 2023-02-14 15:59:26,701 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d736737d4ae8d06714c55079f6c2b15 columnFamilyName cf 2023-02-14 15:59:26,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:26,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:26,702 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:26,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,702 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390366702"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390366702"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390366702"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390366702"}]},"ts":"1676390366702"} 2023-02-14 15:59:26,704 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, ASSIGN in 351 msec 2023-02-14 15:59:26,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:26,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 69a0168d9f604b96943e74817bde200a; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58850068, jitterRate=-0.12306565046310425}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:26,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:26,706 DEBUG [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(539): loaded hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a 2023-02-14 15:59:26,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-02-14 15:59:26,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113 in 201 msec 2023-02-14 15:59:26,706 INFO [StoreOpener-0d736737d4ae8d06714c55079f6c2b15-1] regionserver.HStore(310): Store=0d736737d4ae8d06714c55079f6c2b15/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 15:59:26,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a., pid=266, masterSystemTime=1676390366660 2023-02-14 15:59:26,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, ASSIGN in 356 msec 2023-02-14 15:59:26,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:26,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:26,710 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:26,710 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366710"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390366710"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390366710"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390366710"}]},"ts":"1676390366710"} 2023-02-14 15:59:26,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:26,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0d736737d4ae8d06714c55079f6c2b15; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70889351, jitterRate=0.056333646178245544}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 15:59:26,712 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:26,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15., pid=268, masterSystemTime=1676390366661 2023-02-14 15:59:26,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:26,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:26,721 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:26,721 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390366721"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676390366721"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676390366721"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676390366721"}]},"ts":"1676390366721"} 2023-02-14 15:59:26,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-14 15:59:26,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069 in 204 msec 2023-02-14 15:59:26,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, ASSIGN in 371 msec 2023-02-14 15:59:26,726 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=258 2023-02-14 15:59:26,727 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=258, state=SUCCESS; OpenRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146 in 212 msec 2023-02-14 15:59:26,729 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=258, resume processing ppid=256 2023-02-14 15:59:26,729 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, ASSIGN in 376 msec 2023-02-14 15:59:26,729 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390366729"}]},"ts":"1676390366729"} 2023-02-14 15:59:26,730 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=ENABLED in hbase:meta 2023-02-14 15:59:26,733 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676390362465' was successfully enabled. 2023-02-14 15:59:26,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676390362465 in 390 msec 2023-02-14 15:59:27,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-14 15:59:27,099 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 256 completed 2023-02-14 15:59:27,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:27,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37877] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:27,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:27,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40531] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:27,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:27,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43707] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-14 15:59:27,145 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4672389c to 127.0.0.1:55782 2023-02-14 15:59:27,145 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:27,148 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676390362465 2023-02-14 15:59:27,148 DEBUG [Listener at localhost/39279] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:27,148 INFO [Listener at localhost/39279] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 15:59:27,304 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-14 15:59:27,405 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-14 15:59:27,552 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 15:59:27,603 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-14 15:59:27,734 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-14 15:59:27,922 DEBUG [Listener at localhost/39279] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-14 15:59:29,780 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: d4276e31c48f43eaa70f0eb7fdfc7f6a in the real (hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a) or archive (hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a) directory for the primary table. 2023-02-14 15:59:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: d4276e31c48f43eaa70f0eb7fdfc7f6a in the real (hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a) or archive (hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2506) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2502) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 15:59:29,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] ipc.CallRunner(144): callId: 368 service: MasterService methodName: RestoreSnapshot size: 115 connection: 172.31.14.131:54210 deadline: 1676390429768, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: d4276e31c48f43eaa70f0eb7fdfc7f6a in the real (hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a) or archive (hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/d4276e31c48f43eaa70f0eb7fdfc7f6a) directory for the primary table. 2023-02-14 15:59:29,793 INFO [Listener at localhost/39279] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676390362465 2023-02-14 15:59:29,794 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676390362465 2023-02-14 15:59:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:29,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-14 15:59:29,799 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390369799"}]},"ts":"1676390369799"} 2023-02-14 15:59:29,800 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=DISABLING in hbase:meta 2023-02-14 15:59:29,808 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676390362465 to state=DISABLING 2023-02-14 15:59:29,809 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN}] 2023-02-14 15:59:29,811 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN 2023-02-14 15:59:29,811 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN 2023-02-14 15:59:29,811 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN 2023-02-14 15:59:29,812 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN 2023-02-14 15:59:29,812 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN 2023-02-14 15:59:29,812 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:29,812 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390369812"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390369812"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390369812"}]},"ts":"1676390369812"} 2023-02-14 15:59:29,813 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:29,813 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:29,813 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390369813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390369813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390369813"}]},"ts":"1676390369813"} 2023-02-14 15:59:29,813 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390369813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390369813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390369813"}]},"ts":"1676390369813"} 2023-02-14 15:59:29,813 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:29,814 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390369813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390369813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390369813"}]},"ts":"1676390369813"} 2023-02-14 15:59:29,813 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:29,814 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390369813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390369813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390369813"}]},"ts":"1676390369813"} 2023-02-14 15:59:29,815 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=276, state=RUNNABLE; CloseRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:29,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=275, state=RUNNABLE; CloseRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:29,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=274, state=RUNNABLE; CloseRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:29,818 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=272, state=RUNNABLE; CloseRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146}] 2023-02-14 15:59:29,819 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN 2023-02-14 15:59:29,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=273, state=RUNNABLE; CloseRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069}] 2023-02-14 15:59:29,822 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:29,822 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390369822"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676390369822"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676390369822"}]},"ts":"1676390369822"} 2023-02-14 15:59:29,824 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113}] 2023-02-14 15:59:29,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:29,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c064beb61994c28c74884e236a35d32b, disabling compactions & flushes 2023-02-14 15:59:29,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:29,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:29,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. after waiting 0 ms 2023-02-14 15:59:29,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:29,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c064beb61994c28c74884e236a35d32b 1/1 column families, dataSize=22.24 KB heapSize=48.06 KB 2023-02-14 15:59:29,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:29,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7b10d6f25ee192486c2eccbdfd488bbb, disabling compactions & flushes 2023-02-14 15:59:29,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:29,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:29,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. after waiting 0 ms 2023-02-14 15:59:29,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:29,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7b10d6f25ee192486c2eccbdfd488bbb 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-14 15:59:29,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:29,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0d736737d4ae8d06714c55079f6c2b15, disabling compactions & flushes 2023-02-14 15:59:29,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:29,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:29,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. after waiting 0 ms 2023-02-14 15:59:29,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:29,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0d736737d4ae8d06714c55079f6c2b15 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-14 15:59:30,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-14 15:59:30,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302144768ca46bd424649afa98c3e5f742012 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b202302144768ca46bd424649afa98c3e5f742012 2023-02-14 15:59:30,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021417c43b4c022f4838b84f5b5431e132e0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c2023021417c43b4c022f4838b84f5b5431e132e0 2023-02-14 15:59:30,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230214bc4d2cd8fbf74086bf00782f11e2c7de to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214bc4d2cd8fbf74086bf00782f11e2c7de 2023-02-14 15:59:30,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/.tmp/cf/1f93918f7f844bdf90741aa202d0c1d6 2023-02-14 15:59:30,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/.tmp/cf/2317917edb6543a4ac01514080fb8b31 2023-02-14 15:59:30,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/.tmp/cf/1f93918f7f844bdf90741aa202d0c1d6 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/1f93918f7f844bdf90741aa202d0c1d6 2023-02-14 15:59:30,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/.tmp/cf/2317917edb6543a4ac01514080fb8b31 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/2317917edb6543a4ac01514080fb8b31 2023-02-14 15:59:30,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/1f93918f7f844bdf90741aa202d0c1d6, entries=340, sequenceid=15, filesize=63.5 K 2023-02-14 15:59:30,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.24 KB/22776, heapSize ~48.05 KB/49200, currentSize=0 B/0 for c064beb61994c28c74884e236a35d32b in 220ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:30,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/2317917edb6543a4ac01514080fb8b31, entries=39, sequenceid=15, filesize=11.6 K 2023-02-14 15:59:30,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:30,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b. 2023-02-14 15:59:30,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 0d736737d4ae8d06714c55079f6c2b15 in 233ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:30,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c064beb61994c28c74884e236a35d32b: 2023-02-14 15:59:30,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:30,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:30,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 69a0168d9f604b96943e74817bde200a, disabling compactions & flushes 2023-02-14 15:59:30,207 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=c064beb61994c28c74884e236a35d32b, regionState=CLOSED 2023-02-14 15:59:30,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:30,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:30,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. after waiting 0 ms 2023-02-14 15:59:30,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:30,207 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390370207"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390370207"}]},"ts":"1676390370207"} 2023-02-14 15:59:30,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 69a0168d9f604b96943e74817bde200a 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-14 15:59:30,218 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=276 2023-02-14 15:59:30,218 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=276, state=SUCCESS; CloseRegionProcedure c064beb61994c28c74884e236a35d32b, server=jenkins-hbase4.apache.org,43707,1676390331069 in 395 msec 2023-02-14 15:59:30,221 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=c064beb61994c28c74884e236a35d32b, UNASSIGN in 409 msec 2023-02-14 15:59:30,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:30,244 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15. 2023-02-14 15:59:30,244 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0d736737d4ae8d06714c55079f6c2b15: 2023-02-14 15:59:30,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:30,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:30,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7ae37dd64d13c37bf459120525893ee5, disabling compactions & flushes 2023-02-14 15:59:30,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:30,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:30,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. after waiting 0 ms 2023-02-14 15:59:30,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:30,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7ae37dd64d13c37bf459120525893ee5 1/1 column families, dataSize=1.37 KB heapSize=3.20 KB 2023-02-14 15:59:30,248 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=0d736737d4ae8d06714c55079f6c2b15, regionState=CLOSED 2023-02-14 15:59:30,248 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390370248"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390370248"}]},"ts":"1676390370248"} 2023-02-14 15:59:30,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=272 2023-02-14 15:59:30,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=272, state=SUCCESS; CloseRegionProcedure 0d736737d4ae8d06714c55079f6c2b15, server=jenkins-hbase4.apache.org,37877,1676390331146 in 432 msec 2023-02-14 15:59:30,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=0d736737d4ae8d06714c55079f6c2b15, UNASSIGN in 448 msec 2023-02-14 15:59:30,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230214c3e3684ce677483a94998ca1564af405 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214c3e3684ce677483a94998ca1564af405 2023-02-14 15:59:30,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/.tmp/cf/26fe3dc212ad48508406d2aa6c0c120c 2023-02-14 15:59:30,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/.tmp/cf/26fe3dc212ad48508406d2aa6c0c120c as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/26fe3dc212ad48508406d2aa6c0c120c 2023-02-14 15:59:30,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230214b49943fbbea446318a145b309d7078fb to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214b49943fbbea446318a145b309d7078fb 2023-02-14 15:59:30,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/26fe3dc212ad48508406d2aa6c0c120c, entries=27, sequenceid=15, filesize=9.6 K 2023-02-14 15:59:30,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 69a0168d9f604b96943e74817bde200a in 194ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:30,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:30,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a. 2023-02-14 15:59:30,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 69a0168d9f604b96943e74817bde200a: 2023-02-14 15:59:30,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:30,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/.tmp/cf/6f4beea13c8a40389b459fc21d886589 2023-02-14 15:59:30,433 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=69a0168d9f604b96943e74817bde200a, regionState=CLOSED 2023-02-14 15:59:30,433 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390370433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390370433"}]},"ts":"1676390370433"} 2023-02-14 15:59:30,437 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=273 2023-02-14 15:59:30,437 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=273, state=SUCCESS; CloseRegionProcedure 69a0168d9f604b96943e74817bde200a, server=jenkins-hbase4.apache.org,43707,1676390331069 in 613 msec 2023-02-14 15:59:30,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=69a0168d9f604b96943e74817bde200a, UNASSIGN in 628 msec 2023-02-14 15:59:30,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/.tmp/cf/6f4beea13c8a40389b459fc21d886589 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/6f4beea13c8a40389b459fc21d886589 2023-02-14 15:59:30,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/6f4beea13c8a40389b459fc21d886589, entries=21, sequenceid=15, filesize=8.5 K 2023-02-14 15:59:30,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.37 KB/1405, heapSize ~3.19 KB/3264, currentSize=0 B/0 for 7ae37dd64d13c37bf459120525893ee5 in 201ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:30,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:30,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5. 2023-02-14 15:59:30,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7ae37dd64d13c37bf459120525893ee5: 2023-02-14 15:59:30,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:30,460 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=7ae37dd64d13c37bf459120525893ee5, regionState=CLOSED 2023-02-14 15:59:30,460 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390370460"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390370460"}]},"ts":"1676390370460"} 2023-02-14 15:59:30,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=274 2023-02-14 15:59:30,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=274, state=SUCCESS; CloseRegionProcedure 7ae37dd64d13c37bf459120525893ee5, server=jenkins-hbase4.apache.org,37877,1676390331146 in 643 msec 2023-02-14 15:59:30,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7ae37dd64d13c37bf459120525893ee5, UNASSIGN in 655 msec 2023-02-14 15:59:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-14 15:59:30,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/.tmp/cf/35f2ebf254bb4b9392c996e1d9eea3fe 2023-02-14 15:59:30,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/.tmp/cf/35f2ebf254bb4b9392c996e1d9eea3fe as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/35f2ebf254bb4b9392c996e1d9eea3fe 2023-02-14 15:59:30,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/35f2ebf254bb4b9392c996e1d9eea3fe, entries=33, sequenceid=15, filesize=10.6 K 2023-02-14 15:59:30,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 7b10d6f25ee192486c2eccbdfd488bbb in 618ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:30,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:30,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb. 2023-02-14 15:59:30,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7b10d6f25ee192486c2eccbdfd488bbb: 2023-02-14 15:59:30,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:30,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:30,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 10c0bb63b1c83f26d2954d286d39b71f, disabling compactions & flushes 2023-02-14 15:59:30,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:30,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:30,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. after waiting 0 ms 2023-02-14 15:59:30,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:30,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 10c0bb63b1c83f26d2954d286d39b71f 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-14 15:59:30,598 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=7b10d6f25ee192486c2eccbdfd488bbb, regionState=CLOSED 2023-02-14 15:59:30,599 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676390370598"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390370598"}]},"ts":"1676390370598"} 2023-02-14 15:59:30,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=275 2023-02-14 15:59:30,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=275, state=SUCCESS; CloseRegionProcedure 7b10d6f25ee192486c2eccbdfd488bbb, server=jenkins-hbase4.apache.org,40531,1676390331113 in 785 msec 2023-02-14 15:59:30,603 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=7b10d6f25ee192486c2eccbdfd488bbb, UNASSIGN in 793 msec 2023-02-14 15:59:30,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023021470b0a8a69fda46bd9fda6c7f1e81152b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e2023021470b0a8a69fda46bd9fda6c7f1e81152b 2023-02-14 15:59:30,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/.tmp/cf/15b25e013ee246ac8788c2418827a81c 2023-02-14 15:59:30,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/.tmp/cf/15b25e013ee246ac8788c2418827a81c as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/15b25e013ee246ac8788c2418827a81c 2023-02-14 15:59:30,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/15b25e013ee246ac8788c2418827a81c, entries=40, sequenceid=15, filesize=11.8 K 2023-02-14 15:59:30,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 10c0bb63b1c83f26d2954d286d39b71f in 40ms, sequenceid=15, compaction requested=false 2023-02-14 15:59:30,640 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-14 15:59:30,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f. 2023-02-14 15:59:30,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 10c0bb63b1c83f26d2954d286d39b71f: 2023-02-14 15:59:30,643 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=10c0bb63b1c83f26d2954d286d39b71f, regionState=CLOSED 2023-02-14 15:59:30,643 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676390370643"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676390370643"}]},"ts":"1676390370643"} 2023-02-14 15:59:30,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:30,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-14 15:59:30,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure 10c0bb63b1c83f26d2954d286d39b71f, server=jenkins-hbase4.apache.org,40531,1676390331113 in 820 msec 2023-02-14 15:59:30,646 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=271, resume processing ppid=270 2023-02-14 15:59:30,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676390362465, region=10c0bb63b1c83f26d2954d286d39b71f, UNASSIGN in 836 msec 2023-02-14 15:59:30,647 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676390370647"}]},"ts":"1676390370647"} 2023-02-14 15:59:30,648 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676390362465, state=DISABLED in hbase:meta 2023-02-14 15:59:30,650 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676390362465 to state=DISABLED 2023-02-14 15:59:30,651 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676390362465 in 856 msec 2023-02-14 15:59:31,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-14 15:59:31,303 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 270 completed 2023-02-14 15:59:31,303 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,306 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,306 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-14 15:59:31,312 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:31,312 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:31,312 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:31,312 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:31,312 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:31,312 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:31,314 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits] 2023-02-14 15:59:31,314 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits] 2023-02-14 15:59:31,314 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits] 2023-02-14 15:59:31,314 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits] 2023-02-14 15:59:31,315 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits] 2023-02-14 15:59:31,315 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf, FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits] 2023-02-14 15:59:31,319 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/319d737998834da7ad9f99a4683cfdc2 2023-02-14 15:59:31,319 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/26fe3dc212ad48508406d2aa6c0c120c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/26fe3dc212ad48508406d2aa6c0c120c 2023-02-14 15:59:31,319 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/55a86829b0f24b879d7ddccf917a6091 2023-02-14 15:59:31,320 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/1f93918f7f844bdf90741aa202d0c1d6 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/1f93918f7f844bdf90741aa202d0c1d6 2023-02-14 15:59:31,320 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/2317917edb6543a4ac01514080fb8b31 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/cf/2317917edb6543a4ac01514080fb8b31 2023-02-14 15:59:31,320 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/15b25e013ee246ac8788c2418827a81c to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/15b25e013ee246ac8788c2418827a81c 2023-02-14 15:59:31,321 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/6f4beea13c8a40389b459fc21d886589 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/cf/6f4beea13c8a40389b459fc21d886589 2023-02-14 15:59:31,321 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/cf/c6cef7dc49e2452f8c44360784c16493 2023-02-14 15:59:31,321 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/cf/e0e4bbcdc1a04531af790ce69c79563e 2023-02-14 15:59:31,322 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/35f2ebf254bb4b9392c996e1d9eea3fe to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/cf/35f2ebf254bb4b9392c996e1d9eea3fe 2023-02-14 15:59:31,322 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/cf/41125881fafa49fbb0fe6c39d7da77fb 2023-02-14 15:59:31,324 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15/recovered.edits/18.seqid 2023-02-14 15:59:31,325 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5/recovered.edits/18.seqid 2023-02-14 15:59:31,325 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f/recovered.edits/18.seqid 2023-02-14 15:59:31,325 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b/recovered.edits/18.seqid 2023-02-14 15:59:31,325 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/0d736737d4ae8d06714c55079f6c2b15 2023-02-14 15:59:31,325 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a/recovered.edits/18.seqid 2023-02-14 15:59:31,325 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits/18.seqid to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb/recovered.edits/18.seqid 2023-02-14 15:59:31,326 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7ae37dd64d13c37bf459120525893ee5 2023-02-14 15:59:31,326 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/10c0bb63b1c83f26d2954d286d39b71f 2023-02-14 15:59:31,326 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/c064beb61994c28c74884e236a35d32b 2023-02-14 15:59:31,326 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/69a0168d9f604b96943e74817bde200a 2023-02-14 15:59:31,326 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/.tmp/data/default/testCorruptedSnapshot-1676390362465/7b10d6f25ee192486c2eccbdfd488bbb 2023-02-14 15:59:31,326 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676390362465 regions 2023-02-14 15:59:31,327 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee 2023-02-14 15:59:31,327 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf] 2023-02-14 15:59:31,330 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c2023021417c43b4c022f4838b84f5b5431e132e0 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c2023021417c43b4c022f4838b84f5b5431e132e0 2023-02-14 15:59:31,331 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c202302143ccf38443fb144958318754b8a6f8693 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/a87ff679a2f3e71d9181a67b7542122c202302143ccf38443fb144958318754b8a6f8693 2023-02-14 15:59:31,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b202302144768ca46bd424649afa98c3e5f742012 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b202302144768ca46bd424649afa98c3e5f742012 2023-02-14 15:59:31,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b20230214f2d257aa345040b4818473289f6801fa to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c4ca4238a0b923820dcc509a6f75849b20230214f2d257aa345040b4818473289f6801fa 2023-02-14 15:59:31,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214871a60efa2e340b590b0521d8fc5ca5f to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214871a60efa2e340b590b0521d8fc5ca5f 2023-02-14 15:59:31,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214c3e3684ce677483a94998ca1564af405 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/c81e728d9d4c2f636f067f89cc14862c20230214c3e3684ce677483a94998ca1564af405 2023-02-14 15:59:31,336 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e202302146b061d5d61164507a29090bb79b300ad to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e202302146b061d5d61164507a29090bb79b300ad 2023-02-14 15:59:31,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e2023021470b0a8a69fda46bd9fda6c7f1e81152b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/d41d8cd98f00b204e9800998ecf8427e2023021470b0a8a69fda46bd9fda6c7f1e81152b 2023-02-14 15:59:31,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214bc4d2cd8fbf74086bf00782f11e2c7de to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214bc4d2cd8fbf74086bf00782f11e2c7de 2023-02-14 15:59:31,339 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214ebb5b31f9c904432a43804d8daf83134 to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/e4da3b7fbbce2345d7772b0674a318d520230214ebb5b31f9c904432a43804d8daf83134 2023-02-14 15:59:31,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214628c28ba880c473db782fae69e0c437b to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214628c28ba880c473db782fae69e0c437b 2023-02-14 15:59:31,341 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214b49943fbbea446318a145b309d7078fb to hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/archive/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230214b49943fbbea446318a145b309d7078fb 2023-02-14 15:59:31,342 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/mobdir/data/default/testCorruptedSnapshot-1676390362465/ab001f19dff30b4981c28ca7485899ee 2023-02-14 15:59:31,343 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,347 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676390362465 from hbase:meta 2023-02-14 15:59:31,349 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676390362465' descriptor. 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676390362465' from region states. 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390371350"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390371350"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390371350"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390371350"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390371350"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676390371350"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,352 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-14 15:59:31,352 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 10c0bb63b1c83f26d2954d286d39b71f, NAME => 'testCorruptedSnapshot-1676390362465,,1676390362468.10c0bb63b1c83f26d2954d286d39b71f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 0d736737d4ae8d06714c55079f6c2b15, NAME => 'testCorruptedSnapshot-1676390362465,1,1676390362468.0d736737d4ae8d06714c55079f6c2b15.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 69a0168d9f604b96943e74817bde200a, NAME => 'testCorruptedSnapshot-1676390362465,2,1676390362468.69a0168d9f604b96943e74817bde200a.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7ae37dd64d13c37bf459120525893ee5, NAME => 'testCorruptedSnapshot-1676390362465,3,1676390362468.7ae37dd64d13c37bf459120525893ee5.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7b10d6f25ee192486c2eccbdfd488bbb, NAME => 'testCorruptedSnapshot-1676390362465,4,1676390362468.7b10d6f25ee192486c2eccbdfd488bbb.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c064beb61994c28c74884e236a35d32b, NAME => 'testCorruptedSnapshot-1676390362465,5,1676390362468.c064beb61994c28c74884e236a35d32b.', STARTKEY => '5', ENDKEY => ''}] 2023-02-14 15:59:31,352 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676390362465' as deleted. 2023-02-14 15:59:31,352 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676390362465","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676390371352"}]},"ts":"9223372036854775807"} 2023-02-14 15:59:31,353 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676390362465 state from META 2023-02-14 15:59:31,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 2023-02-14 15:59:31,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676390362465 in 51 msec 2023-02-14 15:59:31,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-14 15:59:31,558 INFO [Listener at localhost/39279] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676390362465, procId: 283 completed 2023-02-14 15:59:31,562 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676390362465" 2023-02-14 15:59:31,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676390362465 2023-02-14 15:59:31,564 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676390362465" 2023-02-14 15:59:31,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33899] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676390362465 2023-02-14 15:59:31,582 INFO [Listener at localhost/39279] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=411 (was 412), OpenFileDescriptor=790 (was 783) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=436 (was 328) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4039 (was 4449) 2023-02-14 15:59:31,582 INFO [Listener at localhost/39279] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-14 15:59:31,582 INFO [Listener at localhost/39279] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-14 15:59:31,582 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x297db6e1 to 127.0.0.1:55782 2023-02-14 15:59:31,583 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,583 DEBUG [Listener at localhost/39279] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-14 15:59:31,584 DEBUG [Listener at localhost/39279] util.JVMClusterUtil(257): Found active master hash=1049046317, stopped=false 2023-02-14 15:59:31,584 INFO [Listener at localhost/39279] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:59:31,588 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 15:59:31,588 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 15:59:31,588 INFO [Listener at localhost/39279] procedure2.ProcedureExecutor(629): Stopping 2023-02-14 15:59:31,588 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 15:59:31,588 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 15:59:31,588 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 15:59:31,589 DEBUG [Listener at localhost/39279] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4325d116 to 127.0.0.1:55782 2023-02-14 15:59:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:59:31,589 DEBUG [Listener at localhost/39279] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:59:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:59:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 15:59:31,589 INFO [Listener at localhost/39279] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43707,1676390331069' ***** 2023-02-14 15:59:31,589 INFO [Listener at localhost/39279] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 15:59:31,589 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37877,1676390331146' ***** 2023-02-14 15:59:31,590 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-14 15:59:31,589 INFO [Listener at localhost/39279] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40531,1676390331113' ***** 2023-02-14 15:59:31,590 INFO [Listener at localhost/39279] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 15:59:31,590 INFO [RS:0;jenkins-hbase4:43707] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 15:59:31,590 INFO [RS:1;jenkins-hbase4:40531] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 15:59:31,590 INFO [RS:0;jenkins-hbase4:43707] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 15:59:31,590 INFO [RS:0;jenkins-hbase4:43707] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 15:59:31,590 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 15:59:31,590 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 15:59:31,590 INFO [RS:1;jenkins-hbase4:40531] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 15:59:31,590 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:31,590 INFO [RS:1;jenkins-hbase4:40531] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 15:59:31,590 DEBUG [RS:0;jenkins-hbase4:43707] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38688fe2 to 127.0.0.1:55782 2023-02-14 15:59:31,590 DEBUG [RS:0;jenkins-hbase4:43707] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,591 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(3304): Received CLOSE for 151fcdb2b3eadef830a0e082099efff3 2023-02-14 15:59:31,591 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43707,1676390331069; all regions closed. 2023-02-14 15:59:31,591 INFO [RS:2;jenkins-hbase4:37877] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 15:59:31,591 INFO [RS:2;jenkins-hbase4:37877] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 15:59:31,591 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 15:59:31,591 INFO [RS:2;jenkins-hbase4:37877] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 15:59:31,591 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:31,591 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 151fcdb2b3eadef830a0e082099efff3, disabling compactions & flushes 2023-02-14 15:59:31,591 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:31,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:59:31,591 DEBUG [RS:2;jenkins-hbase4:37877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x361a42ab to 127.0.0.1:55782 2023-02-14 15:59:31,591 DEBUG [RS:1;jenkins-hbase4:40531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0033bdb6 to 127.0.0.1:55782 2023-02-14 15:59:31,591 DEBUG [RS:2;jenkins-hbase4:37877] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,591 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:59:31,591 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37877,1676390331146; all regions closed. 2023-02-14 15:59:31,591 DEBUG [RS:1;jenkins-hbase4:40531] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,591 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. after waiting 0 ms 2023-02-14 15:59:31,591 INFO [RS:1;jenkins-hbase4:40531] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 15:59:31,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:59:31,592 INFO [RS:1;jenkins-hbase4:40531] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 15:59:31,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 151fcdb2b3eadef830a0e082099efff3 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-14 15:59:31,592 INFO [RS:1;jenkins-hbase4:40531] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 15:59:31,592 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-14 15:59:31,592 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-14 15:59:31,592 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1479): Online Regions={151fcdb2b3eadef830a0e082099efff3=hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3., 1588230740=hbase:meta,,1.1588230740} 2023-02-14 15:59:31,592 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 15:59:31,592 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 15:59:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 15:59:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 15:59:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 15:59:31,593 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-14 15:59:31,598 DEBUG [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1505): Waiting on 151fcdb2b3eadef830a0e082099efff3, 1588230740 2023-02-14 15:59:31,606 DEBUG [RS:0;jenkins-hbase4:43707] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs 2023-02-14 15:59:31,606 INFO [RS:0;jenkins-hbase4:43707] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43707%2C1676390331069:(num 1676390332764) 2023-02-14 15:59:31,606 DEBUG [RS:0;jenkins-hbase4:43707] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,606 DEBUG [RS:2;jenkins-hbase4:37877] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs 2023-02-14 15:59:31,606 INFO [RS:0;jenkins-hbase4:43707] regionserver.LeaseManager(133): Closed leases 2023-02-14 15:59:31,606 INFO [RS:2;jenkins-hbase4:37877] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37877%2C1676390331146:(num 1676390332761) 2023-02-14 15:59:31,606 DEBUG [RS:2;jenkins-hbase4:37877] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,606 INFO [RS:2;jenkins-hbase4:37877] regionserver.LeaseManager(133): Closed leases 2023-02-14 15:59:31,608 INFO [RS:0;jenkins-hbase4:43707] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-14 15:59:31,608 INFO [RS:2;jenkins-hbase4:37877] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 15:59:31,609 INFO [RS:2;jenkins-hbase4:37877] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 15:59:31,609 INFO [RS:0;jenkins-hbase4:43707] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 15:59:31,609 INFO [RS:0;jenkins-hbase4:43707] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 15:59:31,609 INFO [RS:0;jenkins-hbase4:43707] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 15:59:31,609 INFO [RS:2;jenkins-hbase4:37877] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 15:59:31,609 INFO [RS:2;jenkins-hbase4:37877] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 15:59:31,609 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 15:59:31,609 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 15:59:31,614 INFO [RS:0;jenkins-hbase4:43707] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43707 2023-02-14 15:59:31,620 INFO [RS:2;jenkins-hbase4:37877] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37877 2023-02-14 15:59:31,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/.tmp/info/3b979c3872dd436fa3634f4a7ff9f02c 2023-02-14 15:59:31,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/.tmp/info/3b979c3872dd436fa3634f4a7ff9f02c as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/info/3b979c3872dd436fa3634f4a7ff9f02c 2023-02-14 15:59:31,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/info/3b979c3872dd436fa3634f4a7ff9f02c, entries=2, sequenceid=6, filesize=4.8 K 2023-02-14 15:59:31,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 151fcdb2b3eadef830a0e082099efff3 in 35ms, sequenceid=6, compaction requested=false 2023-02-14 15:59:31,635 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/.tmp/info/8b78d8811bc84c0a95688de134cd1b42 2023-02-14 15:59:31,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/namespace/151fcdb2b3eadef830a0e082099efff3/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-14 15:59:31,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:59:31,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 151fcdb2b3eadef830a0e082099efff3: 2023-02-14 15:59:31,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676390333158.151fcdb2b3eadef830a0e082099efff3. 2023-02-14 15:59:31,647 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 15:59:31,647 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 15:59:31,647 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 15:59:31,649 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8b78d8811bc84c0a95688de134cd1b42 2023-02-14 15:59:31,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/.tmp/rep_barrier/33d97fdcf6c14cf083f0a521c1073039 2023-02-14 15:59:31,670 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 33d97fdcf6c14cf083f0a521c1073039 2023-02-14 15:59:31,679 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/.tmp/table/e156122c149c442da3c62882b9cbe09e 2023-02-14 15:59:31,682 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e156122c149c442da3c62882b9cbe09e 2023-02-14 15:59:31,682 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/.tmp/info/8b78d8811bc84c0a95688de134cd1b42 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/info/8b78d8811bc84c0a95688de134cd1b42 2023-02-14 15:59:31,685 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8b78d8811bc84c0a95688de134cd1b42 2023-02-14 15:59:31,686 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/info/8b78d8811bc84c0a95688de134cd1b42, entries=64, sequenceid=307, filesize=11.8 K 2023-02-14 15:59:31,686 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/.tmp/rep_barrier/33d97fdcf6c14cf083f0a521c1073039 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/rep_barrier/33d97fdcf6c14cf083f0a521c1073039 2023-02-14 15:59:31,689 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 33d97fdcf6c14cf083f0a521c1073039 2023-02-14 15:59:31,689 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/rep_barrier/33d97fdcf6c14cf083f0a521c1073039, entries=18, sequenceid=307, filesize=7.0 K 2023-02-14 15:59:31,689 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/.tmp/table/e156122c149c442da3c62882b9cbe09e as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/table/e156122c149c442da3c62882b9cbe09e 2023-02-14 15:59:31,692 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e156122c149c442da3c62882b9cbe09e 2023-02-14 15:59:31,692 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/table/e156122c149c442da3c62882b9cbe09e, entries=22, sequenceid=307, filesize=7.0 K 2023-02-14 15:59:31,693 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 100ms, sequenceid=307, compaction requested=false 2023-02-14 15:59:31,696 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:31,696 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:31,696 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 15:59:31,697 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43707,1676390331069 2023-02-14 15:59:31,697 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@670d8668 rejected from java.util.concurrent.ThreadPoolExecutor@63321e51[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:31,697 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:31,696 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 15:59:31,697 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c6986c9 rejected from java.util.concurrent.ThreadPoolExecutor@63321e51[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:31,697 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@92b87dd rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:31,698 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 15:59:31,697 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 15:59:31,698 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c544b4d rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:31,699 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:31,698 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37877,1676390331146 2023-02-14 15:59:31,699 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@543cbc75 rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:31,699 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37877,1676390331146] 2023-02-14 15:59:31,699 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37877,1676390331146; numProcessing=1 2023-02-14 15:59:31,700 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-14 15:59:31,700 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-14 15:59:31,702 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 15:59:31,702 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-14 15:59:31,702 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-14 15:59:31,702 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37877,1676390331146 already deleted, retry=false 2023-02-14 15:59:31,702 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37877,1676390331146 expired; onlineServers=2 2023-02-14 15:59:31,702 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43707,1676390331069] 2023-02-14 15:59:31,702 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43707,1676390331069; numProcessing=2 2023-02-14 15:59:31,703 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43707,1676390331069 already deleted, retry=false 2023-02-14 15:59:31,703 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43707,1676390331069 expired; onlineServers=1 2023-02-14 15:59:31,798 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40531,1676390331113; all regions closed. 2023-02-14 15:59:31,803 DEBUG [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs 2023-02-14 15:59:31,803 INFO [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40531%2C1676390331113.meta:.meta(num 1676390332891) 2023-02-14 15:59:31,807 DEBUG [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/oldWALs 2023-02-14 15:59:31,807 INFO [RS:1;jenkins-hbase4:40531] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40531%2C1676390331113:(num 1676390332761) 2023-02-14 15:59:31,807 DEBUG [RS:1;jenkins-hbase4:40531] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,807 INFO [RS:1;jenkins-hbase4:40531] regionserver.LeaseManager(133): Closed leases 2023-02-14 15:59:31,807 INFO [RS:1;jenkins-hbase4:40531] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-14 15:59:31,808 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 15:59:31,809 INFO [RS:1;jenkins-hbase4:40531] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40531 2023-02-14 15:59:31,811 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40531,1676390331113 2023-02-14 15:59:31,811 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 15:59:31,812 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40531,1676390331113] 2023-02-14 15:59:31,812 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40531,1676390331113; numProcessing=3 2023-02-14 15:59:31,814 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40531,1676390331113 already deleted, retry=false 2023-02-14 15:59:31,814 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40531,1676390331113 expired; onlineServers=0 2023-02-14 15:59:31,814 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33899,1676390329846' ***** 2023-02-14 15:59:31,814 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-14 15:59:31,815 DEBUG [M:0;jenkins-hbase4:33899] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fba2716, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-14 15:59:31,815 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33899,1676390329846 2023-02-14 15:59:31,815 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33899,1676390329846; all regions closed. 2023-02-14 15:59:31,815 DEBUG [M:0;jenkins-hbase4:33899] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 15:59:31,815 DEBUG [M:0;jenkins-hbase4:33899] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-14 15:59:31,815 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-14 15:59:31,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676390332355] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676390332355,5,FailOnTimeoutGroup] 2023-02-14 15:59:31,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676390332355] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676390332355,5,FailOnTimeoutGroup] 2023-02-14 15:59:31,815 DEBUG [M:0;jenkins-hbase4:33899] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-14 15:59:31,816 INFO [M:0;jenkins-hbase4:33899] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-14 15:59:31,816 INFO [M:0;jenkins-hbase4:33899] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-14 15:59:31,816 INFO [M:0;jenkins-hbase4:33899] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-14 15:59:31,816 DEBUG [M:0;jenkins-hbase4:33899] master.HMaster(1512): Stopping service threads 2023-02-14 15:59:31,817 INFO [M:0;jenkins-hbase4:33899] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-14 15:59:31,817 ERROR [M:0;jenkins-hbase4:33899] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT 2023-02-14 15:59:31,817 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-14 15:59:31,817 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-14 15:59:31,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 15:59:31,818 INFO [M:0;jenkins-hbase4:33899] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-14 15:59:31,818 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-14 15:59:31,818 DEBUG [M:0;jenkins-hbase4:33899] zookeeper.ZKUtil(398): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-14 15:59:31,818 WARN [M:0;jenkins-hbase4:33899] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-14 15:59:31,818 INFO [M:0;jenkins-hbase4:33899] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-14 15:59:31,818 INFO [M:0;jenkins-hbase4:33899] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-14 15:59:31,819 DEBUG [M:0;jenkins-hbase4:33899] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 15:59:31,819 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:59:31,819 DEBUG [M:0;jenkins-hbase4:33899] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:59:31,819 DEBUG [M:0;jenkins-hbase4:33899] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 15:59:31,819 DEBUG [M:0;jenkins-hbase4:33899] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:59:31,819 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=844 KB heapSize=1.01 MB 2023-02-14 15:59:31,834 INFO [M:0;jenkins-hbase4:33899] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=844 KB at sequenceid=2061 (bloomFilter=true), to=hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b1afb68d198e469bbf4673d784dfebc5 2023-02-14 15:59:31,836 INFO [M:0;jenkins-hbase4:33899] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b1afb68d198e469bbf4673d784dfebc5 2023-02-14 15:59:31,837 DEBUG [M:0;jenkins-hbase4:33899] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b1afb68d198e469bbf4673d784dfebc5 as hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b1afb68d198e469bbf4673d784dfebc5 2023-02-14 15:59:31,840 INFO [M:0;jenkins-hbase4:33899] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b1afb68d198e469bbf4673d784dfebc5 2023-02-14 15:59:31,840 INFO [M:0;jenkins-hbase4:33899] regionserver.HStore(1080): Added hdfs://localhost:43519/user/jenkins/test-data/af45deb4-3fbd-6590-b337-5c4f640d8d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b1afb68d198e469bbf4673d784dfebc5, entries=282, sequenceid=2061, filesize=30.7 K 2023-02-14 15:59:31,841 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegion(2947): Finished flush of dataSize ~844 KB/864256, heapSize ~1.01 MB/1056848, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=2061, compaction requested=false 2023-02-14 15:59:31,841 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 15:59:31,841 DEBUG [M:0;jenkins-hbase4:33899] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-14 15:59:31,849 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 15:59:31,849 INFO [M:0;jenkins-hbase4:33899] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-14 15:59:31,850 INFO [M:0;jenkins-hbase4:33899] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33899 2023-02-14 15:59:31,852 DEBUG [M:0;jenkins-hbase4:33899] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,33899,1676390329846 already deleted, retry=false 2023-02-14 15:59:32,088 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,088 INFO [M:0;jenkins-hbase4:33899] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33899,1676390329846; zookeeper connection closed. 2023-02-14 15:59:32,088 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@21b86697 rejected from java.util.concurrent.ThreadPoolExecutor@7730b3f2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,088 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): master:33899-0x100538e544d0000, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,088 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c873c7b rejected from java.util.concurrent.ThreadPoolExecutor@7730b3f2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,188 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,188 INFO [RS:1;jenkins-hbase4:40531] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40531,1676390331113; zookeeper connection closed. 2023-02-14 15:59:32,188 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3ba2ad1b rejected from java.util.concurrent.ThreadPoolExecutor@3f9d3eb7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,188 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:40531-0x100538e544d0002, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,188 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@654b1ed9 rejected from java.util.concurrent.ThreadPoolExecutor@3f9d3eb7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,188 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@46c59e04] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@46c59e04 2023-02-14 15:59:32,288 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,288 INFO [RS:2;jenkins-hbase4:37877] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37877,1676390331146; zookeeper connection closed. 2023-02-14 15:59:32,288 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26395ac0 rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,289 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:37877-0x100538e544d0003, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,289 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@e4ee1ff] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@e4ee1ff 2023-02-14 15:59:32,289 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f38b155 rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,388 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,388 INFO [RS:0;jenkins-hbase4:43707] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43707,1676390331069; zookeeper connection closed. 2023-02-14 15:59:32,389 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@646f39d6 rejected from java.util.concurrent.ThreadPoolExecutor@63321e51[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,389 DEBUG [Listener at localhost/39279-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x100538e544d0001, quorum=127.0.0.1:55782, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 15:59:32,389 ERROR [Listener at localhost/39279-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1690ead8 rejected from java.util.concurrent.ThreadPoolExecutor@63321e51[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 15:59:32,389 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4d30e946] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4d30e946 2023-02-14 15:59:32,389 INFO [Listener at localhost/39279] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-14 15:59:32,389 WARN [Listener at localhost/39279] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 15:59:32,392 INFO [Listener at localhost/39279] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 15:59:32,497 WARN [BP-1677871389-172.31.14.131-1676390326263 heartbeating to localhost/127.0.0.1:43519] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 15:59:32,497 WARN [BP-1677871389-172.31.14.131-1676390326263 heartbeating to localhost/127.0.0.1:43519] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1677871389-172.31.14.131-1676390326263 (Datanode Uuid 10989456-81c2-411f-a52d-a50290e1d258) service to localhost/127.0.0.1:43519 2023-02-14 15:59:32,499 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data5/current/BP-1677871389-172.31.14.131-1676390326263] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 15:59:32,499 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data6/current/BP-1677871389-172.31.14.131-1676390326263] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 15:59:32,501 WARN [Listener at localhost/39279] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 15:59:32,505 INFO [Listener at localhost/39279] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 15:59:32,609 WARN [BP-1677871389-172.31.14.131-1676390326263 heartbeating to localhost/127.0.0.1:43519] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 15:59:32,609 WARN [BP-1677871389-172.31.14.131-1676390326263 heartbeating to localhost/127.0.0.1:43519] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1677871389-172.31.14.131-1676390326263 (Datanode Uuid ce1697e0-c232-403e-a1bb-879864aa955f) service to localhost/127.0.0.1:43519 2023-02-14 15:59:32,612 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data3/current/BP-1677871389-172.31.14.131-1676390326263] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 15:59:32,613 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data4/current/BP-1677871389-172.31.14.131-1676390326263] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 15:59:32,615 WARN [Listener at localhost/39279] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 15:59:32,626 INFO [Listener at localhost/39279] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 15:59:32,730 WARN [BP-1677871389-172.31.14.131-1676390326263 heartbeating to localhost/127.0.0.1:43519] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 15:59:32,730 WARN [BP-1677871389-172.31.14.131-1676390326263 heartbeating to localhost/127.0.0.1:43519] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1677871389-172.31.14.131-1676390326263 (Datanode Uuid a5a6307c-b6e1-4088-b0f5-35057fac4da6) service to localhost/127.0.0.1:43519 2023-02-14 15:59:32,731 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data1/current/BP-1677871389-172.31.14.131-1676390326263] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 15:59:32,732 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2c7073ca-8aa4-e02a-9418-0bcb00c0c5f0/cluster_7b89bf65-e758-c499-8a02-9d9dee8795c0/dfs/data/data2/current/BP-1677871389-172.31.14.131-1676390326263] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 15:59:32,763 INFO [Listener at localhost/39279] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 15:59:32,906 INFO [Listener at localhost/39279] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-14 15:59:33,042 INFO [Listener at localhost/39279] hbase.HBaseTestingUtility(1293): Minicluster is down